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Mission 

Our  mission  is  to  provide  independent,  relevant,  and  timely  oversight 
of  the  Department  of  Defense  that  supports  the  warfighter;  promotes 
accountability,  integrity,  and  efficiency;  advises  the  Secretary  of 
Defense  and  Congress;  and  informs  the  public. 


Vision 

Our  vision  is  to  be  a  model  oversight  organization  in  the 
Federal  Government  by  leading  change,  speaking  truth, 
and  promoting  excellence — a  diverse  organization, 
working  together  as  one  professional  team,  recognized 
as  leaders  in  our  field. 
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Desired  Outcomes  for  SPY-1  Radar  Sustainment 


August  1,  2016 


Findings  (cont'd) 


Objective 

We  determined  whether  the  performance 
metrics  included  in  the  Navy’s  AN/SPY-1  Phased 
Array  Radar  (SPY-1  radar)  performance-based 
logistics  contracts  appropriately  incentivized 
the  support  contractors.  This  audit  is  the 
second  in  a  series  on  SPY-1  radar  spare  parts. 

The  SPY-1  radar  is  an  advanced,  automatic 
detect  and  track  radar  system.  The 
SPY-1  radar  is  one  of  13  major  subsystems 
in  the  AEGIS  Weapon  System  that  searches, 
detects,  and  tracks  air  and  surface  targets 
to  support  Anti-Air  Warfare  and  Ballistic 
Missile  Defense  missions. 

Findings 

Naval  Supply  Systems  Command  Weapon 
Systems  Support  (NAVSUP  WSS) 
did  not  develop  and  incorporate 
appropriate  performance  metrics  into 
the  performance-based  logistics  contracts 
used  to  sustain  SPY-1  radars.  Specifically, 
the  metrics  did  not  effectively  incentivize 
Lockheed  Martin  and  Raytheon  to  achieve 
Navy  warfighter  requirements  or  reduce  the 
total  ownership  cost  associated  with  the 
327  critical  SPY-1  radar  parts  supported  by 
the  contracts.  This  condition  occurred  because 
NAVSUP  WSS  personnel  did  not  follow  DoD 
guidance  when  developing  the  performance 
metrics.  As  a  result,  supply  support  and  cost 
reduction  objectives  for  SPY-1  radar  parts  were 
not  met.  In  addition,  operational  availability  of 
the  AEGIS  Weapon  System  could  be  adversely 
impacted  if  parts  needed  to  maintain  the 
SPY-1  radars  are  not  transported  to  the 
warfighters  when  needed. 


Furthermore,  NAVSUP  WSS  personnel  did  not  adequately 
assess  the  contractors’  performance  against  established 
metrics.  This  occurred  because  NAVSUP  WSS  did  not  have 
written  procedures  to  evaluate  contractors’  performance 
toward  meeting  the  contract  metrics.  As  a  result, 

NAVSUP  WSS  paid  the  contractors  $18  million  during 
NAVSUP  WSS  performance  reviews  without  deducting 
incentive  fees  for  poor  performance  that  was  not  found 
during  these  reviews. 

Recommendations 

We  recommend  that  the  Commander,  Naval  Supply  Systems 
Command,  require  the  Naval  Supply  Systems  Command 
Weapon  Systems  Support  to  follow  DoD  guidance  when 
developing  the  performance  metrics  incorporated  in  future 
performance-based  logistics  contracts  used  to  sustain  the 
SPY-1  radar.  Specifically,  the  Commander  should: 

•  establish  formal  support  agreements  with  Advance 
Traceability  and  Control  and  the  operational  commands 
used  to  supply  SPY-1  radar  parts  to  fleet  customers; 

•  review  the  readiness  and  sustainment  performance 
history  and  costs  of  the  AEGIS  and  SPY-1  radars,  and  use 
that  data  to  identify  the  difference  between  existing  and 
desired  SPY-1  radar  performance  outcomes; 

•  breakdown  system-level  requirements  into  lower-level 
metrics  that  appropriately  link  contractor  performance 
to  the  accomplishment  of  warfighter  readiness  and 
performance  needs; 

•  establish  written  procedures  that  clearly  describe  the 
process  to  conduct  semiannual  performance  reviews 
for  the  performance-based  logistics  contracts;  and 

•  perform  additional  reviews  of  the  completed 
semiannual  reports  for  contracts  N00104-12-D-ZD21 
and  N00104-13-D-ZD00,  to  determine  if  there  is  a 
change  to  the  amount  of  incentives  the  contractors 
received  and  take  corrective  actions  if  appropriate. 


Visit  us  at  www.dodig.mil 

FOR  OFFICIAL  USE  ONLY 

DODIG-2016-116  (Project  No.  D2015-D000AG-0109.000)  |  i 


WBBM  rVCXTT/^TAT  T  TCC  VSESM  V 
x  vyiv  vyi  x  xvx^ixj  vjux/  v/nx-zx 

Results  in  Brief 

Navy  Needs  to  Establish  Effective  Metrics  to  Achieve 
Desired  Outcomes  for  SPY-1  Radar  Sustainment 


Management  Comments  and 
Our  Response 

The  Commander,  NAVSUP,  addressed  the  specifics  of 
the  recommendation  that  NAVSUP  WSS  consult  with  the 
Navy  stakeholders  when  reevaluating  the  SPY-1  radar’s 
product  support  strategy  and  designing  performance 
metrics  included  in  future  performance-based  logistics 
contracts.  However,  the  Commander  did  not  provide 
the  date  when  those  actions  would  be  completed. 

In  addition,  the  Commander  did  not  adequately 
address  establishing  a  formal  agreement  between 
the  two  NAVSUP  organizations  involved  in  supplying 
SPY-1  radar  parts  to  fleet  customers. 

The  Commander,  NAVSUP,  agreed  to: 


However,  comments  from  the  Commander  did  not 
address  all  the  specifics  of  the  recommendations  to  use 
the  data  to  develop  lower-level  metrics  that  incentivize 
the  contractors  to  deliver  the  desired  SPY-1  radar 
performance  outcomes. 

The  Commander  agreed  to  develop  written  procedures 
for  conducting  semiannual  performance  reviews.  The 
Commander  also  agreed  to  reexamine  the  completed 
reviews  for  accuracy.  However,  the  Commander  did  not 
specify  what  actions  the  contracting  officers  would  take 
in  response  to  the  review  reassessment  results.  We 
request  that  the  Commander,  NAVSUP,  provide  comments 
in  response  to  the  report  by  August  31,  2016.  Please  see 
the  Recommendations  Table  on  the  following  page. 


•  review  the  AEGIS  and  SPY-1  historical  performance 
and  costs  to  assess  operational  readiness; 

•  review  the  PBL  performance  metrics  that 
we  determined  were  not  adequate;  and 

•  assess  whether  the  metrics  need  refinement. 
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Recommendations  Table 


Management 

Recommendations 
Requiring  Comment 

No  Additional 
Comments  Required 

Commander,  Naval  Supply  Systems  Command 

A.l.a,  A.l.b,  A.l.c,  A.l.d, 

A.l.e,  B.l.a,  and  B.l.b 

Please  provide  Management  Comments  by  August  31,  2016. 
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INSPECTOR  GENERAL 

DEPARTMENT  OF  DEFENSE 

4800  MARK  CENTER  DRIVE 
ALEXANDRIA,  VIRGINIA 22350-1 500 


August  1,  2016 


MEMORANDUM  FOR  UNDER  SECRETARY  OF  DEFENSE  ACQUISITION, 


TECHNOLOGY  AND  LOGISTICS 
NAVAL  INSPECTOR  GENERAL 


SUBJECT:  Navy  Needs  to  Establish  Effective  Metrics  to  Achieve  Desired  Outcomes  for 
SPY-1  Radar  Sustainment  (Report  No.  DODIG-2016-116] 

We  are  providing  this  report  for  review  and  comment.  Naval  Supply  Systems  Command 
Weapon  Systems  Support  (NAVSUP  WSS]  did  not  establish  performance  metrics  that 
effectively  incentivize  support  providers  to  achieve  warfighter  requirements  and  reduce 
total  ownership  costs  associated  with  SPY-1  radar  parts.  In  addition,  NAVSUP  WSS  did  not 
adequately  assess  contractors'  performance  against  established  metrics.  As  a  result,  supply 
support  and  cost  reduction  objectives  for  SPY-1  radar  parts  were  not  met.  In  addition, 

NAVSUP  WSS  paid  the  contractors  $18  million  without  deducting  incentive  fees  for  poor 
performance.  We  conducted  this  audit  in  accordance  with  generally  accepted  government 
auditing  standards. 

We  considered  management  comments  on  a  draft  of  this  report  when  preparing  the  final 
report.  DoD  Instruction  7650.03  requires  that  recommendations  be  resolved  promptly.  The 
Commander,  NAVSUP,  addressed  all  specifics  of  Recommendation  A.l.a  but  did  not  provide  the 
date  that  NAVSUP  WSS's  actions  would  be  completed.  However,  the  Commander's  comments 
only  partially  addressed  Recommendations  A.l.b,  A.l.c,  A.l.d,  A.l.e,  B.l.a,  and  B.l.b.  Therefore, 
we  request  the  Commander,  NAVSUP,  provide  additional  comments  to  the  final  report  on 
Recommendations  A.l.a,  A.l.b,  A.l.c,  A.l.d,  A.l.e,  B.l.a,  and  B.l.b  by  August  31,  2016. 

Please  send  a  PDF  file  containing  your  comments  to  audasm@dodig.mil.  Copies  of  your 
comments  must  have  the  actual  signature  of  the  authorizing  official  for  your  organization. 

We  cannot  accept  the  /Signed/  symbol  in  place  of  the  actual  signature.  If  you  arrange  to  send 
classified  comments  electronically,  you  must  send  them  over  the  SECRET  Internet  Protocol 
Router  Network  (SIPRNETJ. 

Comments  provided  to  the  final  report  must  be  marked  and  portion-marked,  as  appropriate, 
in  accordance  with  DoD  Manual  5200.01. 

We  appreciate  the  courtesies  extended  to  the  staff.  Please  direct  questions  to  me  at 
[703]  604-9077  (DSN  664-9077], 


Jacqueline  L.  Wicecarver 

Assistant  Inspector  General 

Acquisition  and  Sustainment  Management 
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Introduction 


Objective 

The  audit  objective  was  to  determine  whether  the  performance  metrics  included 
in  the  Navy's  AN/SPY-1  Phased  Array  Radar  (SPY-1  radar]  performance-based 
logistics  [PBL]  contracts  appropriately  incentivized  the  support  contractors.  This 
is  the  second  in  a  series  of  audits  related  to  the  management  of  SPY-1  radar  spare 
parts.  The  first  report  focused  on  the  SPY-1  radar  spare  parts  requirements 
determination  process  and  inventory  management  practices.  See  Appendix  A  for 
a  discussion  of  the  scope  and  methodology  and  prior  audit  coverage  related  to 
the  objective. 

Background 

The  SPY-1  radar  is  an  advanced,  automatic  detect  and  track  radar  system.  The 
SPY-1  radar  is  installed  on  63  Arleigh  Burke- class  guided-missile  destroyers  and 
22  Ticonderoga-c\ass  cruisers.  It  is  one  of  13  major  subsystems  in  the  AEGIS 
Weapon  System  that  searches,  detects,  and  tracks  air  and  surface  targets  to 
support  Anti-Air  Warfare  and  Ballistic  Missile  Defense  missions.  The  SPY-1  radar 
communicates  with  the  standard  missile,  provides  information  to  guide  the  missile 
to  the  target,  and  assesses  the  missile's  success  in  destroying  the  target.  The  Navy 
placed  the  first  SPY-1  radar  in  operation  in  1983  on  the  USS  Ticonderoga- class 
cruisers  and  on  the  Arleigh  Burke-class  destroyers  in  1991.  Figure  1  illustrates 
the  SPY-1  radar's  capabilities. 
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Figure  1.  AN/SPY-1  Phased  Array  Radar  System  Capabilities 


Source:  Naval  Sea  Systems  Command  Leading  Edge,  Volume  1,  Issue  No.  2 

Naval  Supply  Systems  Command 

The  Naval  Supply  Systems  Command  manages  the  Navy's  supply  system  and 
provides  material  support  for  Navy  surface  ships,  submarines,  aircraft,  and 
expeditionary  forces.  After  a  weapon  system  is  fully  developed  and  integrated  into 
the  fleet,  Naval  Supply  Systems  Command  Weapon  Systems  Support  (NAVSUP  WSS]: 

•  assumes  responsibility  for  supporting  that  system; 

•  provides  the  fleet  with  parts  through  a  multi-tiered  retail  system  and 
wholesale  inventory; 

•  manages  parts  inventory  for  ships,  submarines  and  weapon  systems, 
including  support  for  hull,  electrical,  mechanical,  and  electrical 
components;  and 

•  forecasts  parts  requirements  for  wholesale  stocking. 

Fleet  customers  use  operations  and  maintenance  funding  to  purchase  parts  from 
NAVSUP  WSS  wholesale  inventory.  NAVSUP  WSS  uses  the  wholesale  system  to 
purchase  spare  parts  from  vendors  with  Navy  Working  Capital  Funds,  and  then 
resells  the  parts  to  fleet  customers. 


2  |  DODIG-2016-116 


FOR  OFFICIAL  USE  ONLY 


FOR  OFFICIAL  USE  ONLY 


Introduction 


Performance-Based  Logistics  Arrangements 

The  DoD  designated  performance-based  logistics  (PBL]  as  the  preferred  equipment 
sustainment  strategy  in  an  effort  to  increase  weapon  systems  readiness,  while 
reducing  support  costs  and  supply  chain  infrastructure.  In  2003,  DoD  incorporated 
into  policy  the  requirement  for  acquisition  managers  to  use  performance-based 
arrangements  for  sustaining  products  and  services  wherever  feasible.  A  PBL 
contract  provides  weapon  system  support  by  designating  what  system  performance 
outcome  is  required,  such  as  acquiring  a  level  of  availability,  and  incentivizes  the 
contractor  to  reduce  costs  through  innovation. 

The  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  (USD[AT&L]]  policy  memorandum1  states  that  PBLs  are  a  method 
to  achieve  DoD's  Better  Buying  Power  goals  and  their  appropriate  use  would 
help  to  achieve  affordable  sustainment  strategies.  The  USD(AT&L]  Implementing 
Directive  for  Better  Buying  Power  2.02  states  that  the  history  of  PBL  contracting 
demonstrates  DoD  can  achieve  improved  readiness  at  significant  savings  if  PBL 
business  arrangements  are  properly  structured  and  executed. 

As  a  business  model,  PBL  differs  from  traditional  contract  support  in  many 
ways.  In  a  PBL  arrangement,  the  Government  buys  performance  outcomes.3 
PBL  contracts  do  not  provide  detailed  descriptions  of  what  Government  goods 
and  services  are  required  to  meet  the  outcomes.  Rather,  the  Government 
identifies  its  desired  system  performance  and  the  contractor  determines  how  to 
deliver  that  performance  outcome.  In  a  PBL  sustainment  contract,  the  contractor 
is  incentivized  to  reduce  repairs,  cost  of  parts,  and  labor  because  its  profit  is 
increased  by  reducing  costs. 

In  contrast,  for  a  traditional  support  model,  the  Government  buys  a  weapon  system 
and  associated  spares,  repairs,  tools,  and  data.  The  Government  must  specify 
which  goods  and  services  are  desired  and  how  many  of  each  is  needed.  Traditional 
contracts  focus  on  transactions  of  goods  or  services.  The  contractor  charges  the 
Government  for  each  repair  or  replacement  transaction  when  equipment  fails  or 
requires  overhaul.  As  a  result,  the  contractor  receives  more  revenue  the  more 
the  equipment  fails,  which  is  misaligned  with  the  Government's  goals  for  reliable, 
affordable  equipment.  According  to  a  study  sponsored  by  the  Assistant  Secretary 
of  Defense  (Logistics  &  Materiel  Readiness]  (ASD[L&MR]],4the  Government  should 


1  USD(AT&L)  memorandum,  "Endorsement  of  Next-Generation  Performance-Based  Logistics  Strategies,"  May  14,  2012. 

2  USD(AT&L)  memorandum,  "Implementation  Directive  for  Better  Buying  Power  2.0  -  Achieving  Greater  Efficiency  and 
Productivity  in  Defense  Spending,”  April  24,  2013. 

3  For  PBL,  performance  is  defined  in  terms  of  military  objectives,  such  as  operational  availability,  reliability,  cost  per  unit 
usages,  or  logistics  response  time. 

4  Proof  Point  Project:  A  Study  to  Determine  the  Impact  of  Performance  Based  Logistics  (PBL)  on  Life  Cycle  Costs, 
November  2011. 
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share  in  the  cost  savings  achieved  by  the  contractor.  These  savings  could  occur 
during  the  performance  of  the  existing  contract  or  when  awarding  a  follow-on 
contract  at  a  lower  price  or  both. 

SPY-1  Radar  Performance-Based  Logistics  Contracts 

NAVSUP  WSS  used  two  PBL  contracts  to  support  327  parts  critical  to  the 
sustainment  of  the  SPY-1  radars. 

•  On  August  9,  2012,  NAVSUP  WSS  awarded  a  5-year,  requirements-type, 
sole-source  contract5  to  the  Lockheed  Martin  Corporation  for 

AEGIS  Weapon  System  materiel  support.  As  of  November  26,  2013, 
NAVSUP  WSS  obligated  $11.4  million  to  sustain  the  SPY-1  radars. 

•  On  November  28,  2012,  NAVSUP  WSS  awarded  a  5-year,  requirements-type, 
sole-source  contract6  to  the  Raytheon  Company  to  provide  materiel 
support  for  two  AEGIS  subsystems:  the  MK-99  Fire  Control  System  and 
the  SPY-1  radar  transmitter  group.  During  the  first  year  of  the  contract, 
NAVSUP  WSS  obligated  $16.6  million,  to  sustain  the  SPY-1  radars. 

Desired  Performance  Outcomes 

According  to  the  SPY-1  radar  contract  work  statements,  the  goal  of  the  contracts 
was  to  create  a  flexible,  streamlined  process  between  the  Navy  and  Lockheed 
Martin  and  Raytheon  that  would  reduce: 

•  administrative  lead  time/cycle  time  (logistics  response  time);7 

•  support  process  variation;  and 

•  the  total  ownership  cost  associated  with  the  327  critical  SPY-1  radar  parts 
supported  in  the  contracts. 

Delivery  Requirements 

The  SPY-1  radar  contracts  required  Lockheed  Martin  and  Raytheon  to  fill  customer 
orders  within  specified  timeframes.  Table  1  shows  the  various  requisition  types 
and  number  of  days  specified  in  the  contracts  that  Lockheed  Martin  and  Raytheon 
agreed  to  supply  requested  parts  to  Advanced  Traceability  and  Control  (ATAC)B 
for  shipment. 


5  Lockheed  Martin  contract  number  N00104-12-D-ZD21. 

6  Raytheon  contract  number  N00104-13-D-ZD00. 

7  Logistics  response  time  is  the  average  number  of  days  that  elapses  from  the  time  a  customer  submits  a  requisition 
(order)  to  the  time  the  customer  receives  the  parts  ordered. 

8  ATAC  is  a  NAVSUP  WSS  organization  that  picks  up  and  delivers  parts  to  Navy  customers. 
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Table  1.  Contracted  Delivery  Times 


Order  Priority  Category* 

Delivery  Requirement 

Casualty  Report 

Within  24  Hours 

Category  1  (Priority  Codes  1-3) 

Within  2  Work  Days 

Category  2  (Priority  Codes  4-8) 

Within  4  Work  Days 

Category  3  (Priority  Codes  9-15) 

Within  6  Work  Days 

*  DoD  uses  order  priority  categories  to  convey  how  urgently  customers  require  the  part  ordered.  Priority 
designators  range  from  1  to  15.  Part  orders  designated  priority  group  1  have  a  higher  priority  than  other 
requisitions  designations  and  have  shorter  delivery  requirements. 

Source:  Lockheed  Martin  and  Raytheon  contracts  and  DoD  Regulation  4140. 1-R,  May  23,  2003 


Performance  Metrics 

NAVSUP  WSS  used  three  performance  metrics  to  assess  contractor  success  in  filling 
part  orders.  Table  2  shows  the  metrics  standards,  and  the  negative  or  positive 
adjustments  that  are  applied  to  the  contractors'  next  payment,  based  on  the 
respective  performance  levels.9 


Table  2.  Performance  Metric  and  Payment  Adjustment 


/cm  im 

WWW/ 

PBL 

Contractors 


Lockheed 

Martin 


Average  Fill  Rate 


Average  Contractor 
Response  Time 


Average  Casualty 
Report  Response  Time 


►  r 


Raytheon 


r 


(FOUO) 


*  Performance  Metric  Standard 

Source:  Lockheed  Martin  and  Raytheon  contracts 


9  In  addition,  Lockheed  Martin  incurs  a  $1,250  penalty  for  each  order  that  remains  undelivered  for  more  than  the  agreed 
to  number  of  production  lead-time  days.  Raytheon  incurs  a  $1,000  penalty  for  each  order  that  remains  undelivered  for 
more  than  365  days  and  a  $3,000  penalty  for  each  order  requiring  expedited  delivery  that  remains  undelivered  for  more 
than  9  days. 
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As  of  August  2015,  NAVSUP  WSS  program  management  personnel  had  reviewed 
Lockheed  Martin's  first  year  performance  and  Raytheon's  first  6-month 
performance.10 

Review  of  Internal  Controls 

DoD  Instruction  5010.4011  requires  DoD  organizations  to  implement  a 
comprehensive  system  of  internal  controls  that  provides  reasonable  assurance 
that  programs  are  operating  as  intended  and  to  evaluate  the  effectiveness  of  the 
controls.  We  identified  internal  control  weaknesses  with  the  procedures  that 
NAVSUP  WSS  used  to  develop  the  performance  metrics  in  the  PBL  contracts  to 
sustain  the  SPY-1  radars  and  the  steps  taken  to  assess  contractor  performance 
against  those  metrics.  We  will  provide  a  copy  of  this  report  to  the  senior  official 
responsible  for  internal  controls  in  the  Office  of  the  Chief  of  Naval  Operations. 


10  NAVSUP  WSS  personnel  stated  they  were  behind  on  the  Raytheon  performance  review  because  of  staffing  turnover. 

11  DoD  Instruction  5010.40,  "Managers'  Internal  Control  Program  Procedures,"  May  30,  2013. 
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SPY-1  Radar  System  Performance  Metrics 
Were  Ineffective 

NAVSUP  WSS  did  not  develop  and  incorporate  appropriate  performance  metrics 
into  the  PBL  contracts  that  sustained  SPY-1  radars.  Specifically,  the  contract 
performance  metrics  did  not  effectively  incentivize  Lockheed  Martin  and  Raytheon 
to  achieve  Navy  warfighter  requirements  or  reduce  the  total  ownership  cost12 
associated  with  the  327  critical  SPY-1  radar  parts  supported  by  the  contracts. 

This  occurred  because  NAVSUP  WSS  personnel  did  not  follow  DoD  guidance  when 
developing  the  performance  metrics  in  these  contracts.  As  a  result,  supply  support 
and  total  ownership  cost  reduction  objectives  for  SPY-1  radar  parts  were  not  being 
met.  In  addition,  the  operational  availability  of  the  AEGIS  Weapon  System  could  be 
adversely  impacted  if  parts  needed  to  maintain  the  SPY-1  radars,  a  critical  AEGIS 
subsystem,  are  not  provided  to  the  warfighters  when  needed. 


Performance  Metrics  Not  Developed  to  Incentivize 
Contractors  to  Achieve  Desired  Outcomes 

NAVSUP  WSS  personnel  did  not  develop  and  incorporate  appropriate  performance 
metrics  into  the  PBL  contracts  used  to  sustain  the  SPY-1  radars.  Specifically,  the 
contract  performance  metrics  did  not  effectively  incentivize  Lockheed  Martin  and 
Raytheon  to  achieve  Navy  warfighter  requirements  or  reduce  the  total  ownership 
cost  associated  with  the  327  critical  SPY-1  radar  parts  supported  by  the  contracts. 

DoD  PBL  guidance13  states  that  a  PBL  is  to  create  an  arrangement,  which 
incentivizes  contractors  to  reduce  costs  by  improving  a  weapon  system's 
operational  availability  and  decreasing  maintenance  costs.  According  to  the 
DoD  PBL  guidance,  for  PBLs  to  be  effective,  desired  outcomes  must  have  supporting 
performance  metrics  that  objectively  indicate  whether  desired  outcomes  have  been 
accomplished.  In  addition,  the  DoD  PBL  guidance  states  that  the  PBL  arrangement 
should  be  detailed  in  a  formal  agreement,  clearly  defining  the  desired  outcomes;  the 
associated  performance  metrics  and  targets  desired;  and  the  financial  incentives 
and  disincentives  for  not  meeting  or  exceeding  agreed-upon  desired  outcomes. 


12  Total  ownership  cost  includes  the  direct  costs  of  the  program,  logically  attributable  indirect  program  costs, 
support  costs,  and  the  costs  associated  with  important  central  logistics  infrastructure  activities  (such  as  supply 
chain  management). 

13  DoD  Performance  Based  Logistics  (PBL)  Guidebook,  May  27,  2014,  Product  Support  Manager  Guidebook,  April  2011,  and 
OSD  L&MR  PBL  study. 
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SPY-1  Radar  Product  Support  Strategy 

According  to  PBL  Best  Practices  draft  guidance14,  NAVSUP  WSS  acts  as  the  Product 
Support  Integrator  in  many  product  support15  arrangements.  In  this  capacity, 
NAVSUP  WSS  is  the  program  manager  and  product  support  manager's  agent  for 
sustainment.  Product  support  strategies  can  take  many  forms  that  leverage  the 
capabilities  of  a  variety  of  product  support  providers,  including  both  commercial 
and  Government  sources.  NAVSUP  WSS  used  the  PBL  arrangement  to  implement 
the  product  support  strategy  in  supplying  parts  for  the  SPY-1  radar.  NAVSUP  WSS 
entered  into  an  arrangement  that  assigned  roles,  responsibilities,  and  the 
performance  expectations  of  the  product  support  providers.  Figure  2  shows 
the  process  for  the  product  support  strategy  that  NAVSUP  WSS  used  to  supply  the 
parts  the  fleet  needs  to  sustain  the  SPY-1  radar. 


Figure  2.  SPY-1  Radar  Product  Support  Process 


r  ^ 

1.  A  Customer  Submits 
an  Order  for  a  Part  to 
NAVSUP  WSS 

L _ J 


5.  ATAC  Delivers  Part 
to  Customer 


4.  The  ATAC 
Transportation  Carrier 
Picks  Up  the  Part 


2.  NAVSUP  WSS  Uses 
the  CAV  System  to 
Release  the  Order 
Electronically  to  the 
PBL  Contractor 


V 


"3.  The  PBL  Contractor’ 
Fills  the  Order  and 
Identifies  in  CAV 
System  the  Part  is 
Ready  for  Pick  Up  by 
the  ATAC 

.Transportation  Carrier. 


LEGEND: 

ATAC  Advance  Traceability  and  Control 

CAV  Commercial  Asset  Visibility 

NAVSUP  WSS  Naval  Supply  Systems  Command  Weapon  Systems  Support 
PBL  Performance-Based  Logistics 

Source:  Lockheed  Martin  contract  N00104-12-D-ZD21  and  Raytheon  contract  N00104-13-D-ZD00 


14  NAVSUP  Draft  PBL  Best  Practices  (Steps  8  through  12),  September  2013. 

15  Product  support  includes  the  logistics  support  functions  necessary  to  maintain  the  readiness  and  operational  capability 
of  a  system  or  subsystem.  It  encompasses  a  range  of  disciplines  including,  but  not  limited  to:  logistics,  requirements, 
operational  mission  planning,  financial,  contracts,  legal,  and  integrated  product  support  elements. 
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Contract  Performance  Metrics  Not  Aligned  With  the  Desired 
Outcomes  or  Adjusted  to  the  Product  Support  Strategy 

NAVSUP  WSS  incorporated  the  following  performance  metrics  in  the  Lockheed 
Martin  and  Raytheon  contracts: 


•  Average  Fill  Rate — percentage  of  orders  that  contractors  filled  within  the 
time  frames  specified  in  the  contract; 

•  Average  Contractor  Response  Time — average  time  (in  days]  it  takes  the 
contractor  to  fill  customer  orders;  and 

•  Average  Casualty  Report  Response  Time — average  time  (in  days]  it  takes 
the  contractor  to  fill  urgent  customer  orders  needed  within  24  hours 

of  receipt. 


However,  the  contract  performance  metrics  only  evaluate  the 
success  of  the  contractors'  timeliness  in  making  requested 
parts  available  for  pick  up  by  ATAC.  Without  additional 

i 

metrics  that  are  specifically  designed  to  incentivize 
the  contractors  to  achieve  all  the  contract  goals,16  the 
contractors  will  not  streamline  their  supply  support 
process  or  take  other  steps  that  will  reduce  the  total 
cost  of  supporting  the  SPY-1  radar.  NAVSUP  WSS  did 
not  incorporate  any  contractual  incentives  that  would 
motivate  Lockheed  Martin  and  Raytheon  to  take  actions  to 
accomplish  those  performance  goals. 


The  contract 
performance 
metrics  only  evaluate 
the  success  of  the 
contractors'  timeliness  in 
making  requested  parts 
available  for  pick 
up  by  ATAC. 


In  addition,  NAVSUP  WSS  did  not  incorporate  the  customer-required  delivery  times 
or  the  time  needed  by  ATAC  to  deliver  the  parts  to  customers17  when  developing 
the  performance  metric  standards  in  the  Lockheed  Martin  and  Raytheon 
contracts.  For  example,  if  a  customer  located  in  the  continental  United  States 
submits  a  Category  2  part  order,  the  customer  may  not  receive  the  part  when 
required.  Category  2  part  orders  are  to  be  delivered  to  customers  located  in  the 
continental  United  States  within  10  days.  NAVSUP  WSS  required  Lockheed  Martin 
and  Raytheon  to  make  Category  2  part  orders  available  to  ATAC  within  4  days. 
However,  ATAC  business  rules  state  that  it  takes  8  days  to  deliver  Category  2  part 
orders.  Therefore,  unless  Lockheed  Martin  and  Raytheon  can  make  the  parts 
available  in  2  days,  or  ATAC  delivers  the  parts  to  the  customer  in  6  days,  customers 
will  not  receive  the  requested  parts  when  needed. 


16  The  contract  goals  are  reduce  logistics  response  time,  process  variation,  and  total  ownership  cost. 

17  Customer  delivery  times  vary  based  on  where  the  customer  is  located  and  how  urgently  the  parts  are  needed.  In 
addition,  ATAC  takes  4  to  8  days  to  deliver  required  parts. 
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NAVSUP  WSS  Did  Not  Follow  DoD  Guidance 

NAVSUP  WSS  personnel  did  not  follow  DoD  guidance  when  developing  the 
performance  metrics  in  the  SPY-1  radar  PBL  contracts.  The  objective  of  product 
support  is  to  implement  a  sustainment  strategy  that  delivers  affordable  readiness, 
which  is  defined  as  providing  mission  capability  to  the  warfighter  at  the  lowest 
cost  to  the  taxpayer.  DoD  uses  the  12-step  process18  shown  in  Figure  3  to  develop 
weapon  system  support  strategies,  including  those  that  rely  on  PBLs. 


Figure  3.  DoD  12-Step  Product  Support  Strategy  Model 
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Source:  DoD  Product  Support  Manager  Guidebook,  April  2011 


18  See  Appendix  B  for  actions  that  typically  occur  during  the  12-step  process. 
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However,  NAVSUP  WSS  did  not  use  the  12-step  process  and  instead  incorporated 
the  metrics  that  NAVSUP  WSS  had  always  included  in  maritime  PBL  contracts. 
Specifically,  we  did  not  find  evidence  that  NAVSUP  WSS: 

•  obtained  relevant  subject  matter  expertise  needed  to  translate  system 
requirements  into  a  viable  product  support  strategy  during  the  Product 
Support  Management  Integrated  Product  Support  Team  (IPT]  (Step  2]; 

•  baselined  the  system  to  assess  the  current  product  support  strategy  and 
established  initial  technical  and  cost  and  pricing  baselines  (Step  3];  and 

•  identified  and  refined  the  contract  metrics  to  achieve  desired  performance 
outcomes  (Step  4], 

Relevant  Expertise  Not  Used  to  Develop  Performance  Metrics 

NAVSUP  WSS  did  not  use  relevant  subject  matter  experts  when  reevaluating 
the  SPY-1  radar's  product  support  strategy  and  developing  the  performance 
metrics  included  in  the  Lockheed  Martin  and  Raytheon  PBL  contracts.  Although 
NAVSUP  WSS  used  an  Integrated  Product  Team  (IPT]19  when  reissuing  the  PBL 
contracts,  the  IPT  did  not  include  ATAC  and  fleet  personnel.  Forming  an  IPT  is  an 
important  step  to  developing  an  effective  product  support  strategy  and  requires 
participation  and  consensus  of  all  stakeholders,  including  the  warfighter.  The  war 
fighting  capability  needs  must  be  translated  into  requirements.  The  metrics  are 
derived  from  the  requirements  to  drive  achievement  of  the  desired  outcomes  and 
should  be  documented  in  formal  agreements  and  serve  as  the  primary  measures  of 
support  provider  performance. 

Periodically  obtaining  stakeholders  insight  and  tailoring  performance  metrics 
for  changes  to  the  specific  weapon  system  and  the  operational  environment  is 
critical  to  ensuring  the  product  support  strategy  remains  in  close  alignment  with 
warfighter  requirements.  Developing  a  weapon  system's  product  support  strategy 
is  not  a  one-time  decision  made  early  in  a  system's  life  and  executed  in  the  same 
form  throughout  the  life  cycle.  A  product  strategy  for  a  weapon  system  evolves 
over  time,  since  the  requirements,  capabilities,  competencies,  operational  mission, 
and  material  condition  of  defense  systems  change  over  time. 

We  did  not  find  evidence  that  NAVSUP  WSS  compared  ATAC's  business  processes 
to  the  fleet  customer  delivery  requirements  for  the  various  categories  of  parts 
shipments  and  made  appropriate  revisions.  In  addition,  NAVSUP  WSS  personnel 
stated  that  they  did  not  enter  into  a  formal  agreement  with  ATAC  that  defined 


19  The  IPT  is  a  collaborative  working  body  comprising  key  program  staff  and  stakeholders  whose  purpose  is  to  develop 
a  product  support  solution.  Various  disciplines  should  be  represented  within  the  IPT,  including  Life  Cycle  Logistics, 
Engineering,  Finance,  Contracting,  Legal,  and  individuals  from  other  functional  groups  specific  to  the  program  and 
life-cycle  needs. 
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the  specific  timeframes  that  ATAC  needed  to  follow  to  ensure  that  parts  were 
transported  to  the  fleet  when  needed.  As  a  result,  NAVSUP  WSS  did  not  establish 
appropriate  standards  in  the  contracts  for  Lockheed  Martin  and  Raytheon's 
performance  metrics,  which  contributed  to  parts  not  being  delivered  to  Navy 
customers  timely  using  ATAC  transportation. 

Navy  customers  use  DoD's  order  processing  and  delivery  time  standards  to 
communicate  the  delivery  requirements  to  support  providers.  For  example,  fleet 
customers  used  "Category  1"  to  inform  NAVSUP  WSS  that  a  part's  delivery  needed 
to  be  expedited,  because  the  part  was  urgently  needed.  Fleet  customers  submitted 
190  Category  1  part  orders  during  NAVSUP  WSS  performance  review  periods.20 
The  fleet  requested  expedited  delivery  of  these  parts  because  the  SPY-1  radar  may 
not  perform  its  mission  without  them.  We  determined  that  137  of  those  190  orders 
were  not  delivered  by  the  required  delivery  dates.  See  Appendix  C  for  a  list  of  the 
part  orders  not  filled  by  the  customers'  required  delivery  dates. 

In  addition,  NAVSUP  WSS  did  not  contact  the  fleet  to  determine  whether  the  PBL 
performance  metrics  should  have  changed,  based  on  the  operational  environment 
of  the  SPY-1  radar  at  the  time,  when  NAVSUP  WSS  renewed  the  Lockheed  Martin 
and  Raytheon  PBL  contracts  in  August  2012  and  November  2012,  respectively.  On 
October  1,  2012,  the  Director,  Surface  Warfare  Division,  Chief  of  Naval  Operations, 
issued  an  administrative  change  memorandum21  that  increased  the  operational 
availability  requirement  of  the  AEGIS  Weapon  System  and  the  SPY-1  radars.  In 
the  memorandum,  the  Director  stated: 

With  more  frequent  deployments,  longer  at-sea  time,  and  the 
critical  nature  of  the  ballistic  missile  defense  mission,  AEGIS  ships 
were  unable  to  maintain  the  AEGIS  Combat  Systems  components 
at  required  levels  to  meet  Combatant  Commanders,  Fleet, 
and  Type  Command  requirements.  [A  December  2011  AEGIS 
Readiness  Sparing  study]  suggested  a  major  contributing  factor 
to  the  mission  degradation  was  insufficient  spare  parts  in  ship’s 
inventory.  By  increasing  the  operational  availability  for  use  with 
the  AEGIS  Readiness  Based  Sparing  (RBS)  model,  it  is  anticipated 
that  the  sparing  inventory  on  board  AEGIS  ships  will  increase, 
resulting  in  improved  repair  time  to  meet  mission  requirements. 

[Emphasis  Added], 


20  NAVSUP  WSS  performed  two  semiannual  reviews  covering  Lockheed  Martin's  performance  from  August  10,  2012, 
through  August  9,  2013,  and  one  semiannual  review  covering  Raytheon's  performance  from  November  28,  2012, 
through  May  27,  2013. 

21  Chief  of  Naval  Operations  memorandum,  "Authority  to  Increase  Operational  Availability  (Ao)  for  AEGIS  Sparing," 
October  1,  2012. 
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We  did  not  find  evidence  that  NAVSUP  WSS  ever  assessed  the  impact  that  the 
change  made  to  the  SPY-1  radar's  operational  availability  requirement  had  on  its 
supply  support.  Specifically,  NAVSUP  WSS  did  not  determine  if  the  existing  PBL 
metrics  needed  to  be  modified  because  of  the  December  2011  AEGIS  Readiness 
Sparing  study  findings.  The  study  concluded  that  supply  support  was  not  keeping 
up  with  the  fleet's  operational  availability  requirement  for  the  SPY-1  radar. 

NAVSUP  WSS  should  consult  with  ATAC  and  the  operational  commands  that  support 
the  fleet  when  reevaluating  the  SPY-1  radar's  product  support  strategy  and  designing 
the  performance  metrics  included  in  future  performance-based  logistics  contracts. 

In  addition,  NAVSUP  WSS  should  establish  formal  support  agreements  with  ATAC 
and  the  operational  commands  used  to  supply  SPY-1  radar  parts  to  fleet  customers. 

SPY-1  Radar  Product  Support  Process  Not  Reviewed  and 
Technical  and  Cost  Baselines  Not  Established 

NAVSUP  WSS  did  not  review  the  current  product  support  process  and  establish 
a  baseline22  for  the  logistical  functions  and  total  costs  before  awarding  the 
SPY-1  PBL  contracts.  DoD  guidance23  provides  steps  to  define  and  implement  a 
product  support  strategy  that  is  affordable  and  effective.  The  guidance  explains 
that  baselining  the  system  provides  a  foundation  for  developing  a  strong  product 
support  strategy.  It  states  that  NAVSUP  WSS  should  assess  the  supply  strategy, 
including  monitoring  contractor  performance,  improvements,  affordability,  and 
cost  control. 

Effective  PBL  implementation  depends  on  identifying  and  refining  performance 
metrics  that  accurately  reflect  warfighter  requirements  and  measure  contractor 
performance.  The  ASD(L&MR]  issued  a  memorandum24  stating  that  the 
Government  must  clearly  understand  the  program  requirements,  cost  and 
technical  characteristics,  along  with  associated  tradeoffs  and  alternatives  for 
PBL  arrangements  to  be  effective.  The  memorandum  further  states  one  of  the 
indicators  of  an  effective  PBL  arrangement  is  that  the  Government  gains  data 
during  the  period  of  performance  to  refine  subsequent  PBL  arrangements  for 
improved  productivity  and  cost  reduction.  NAVSUP  WSS  should  have  defined 
the  SPY-1  radar's  existing  performance  and  cost  baselines  as  a  starting  point 
for  monitoring  performance  and  costs  to  ensure  processes  are  achieving 
required  outcomes. 


22  The  process  of  developing  the  system  baseline  is  to  identify  all  of  the  information  known  about  the  system  to  include 
performance,  support,  reliability,  maintainability,  and  cost  data. 

23  DoD  Product  Support  Manager  Guidebook,  April  2011,  provides  guidance  on  how  to  develop  and  execute  a  product 
support  strategy. 

24  ASD(L&MR)  memorandum,  "Performance  Based  Logistics  Comprehensive  Guidance,"  November  22,  2013. 
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Lockheed  Martin  and  Raytheon  provided  logistical  functions  for  these 
PBL  contracts,  including: 

•  making  inventory  decisions; 

•  repairing  and/or  manufacturing  all  parts; 

•  providing  associated  warehousing; 

•  managing  system  configuration  and  parts  obsolescence;  and 

•  coordinating  transportation  and  tracking  part  orders. 

NAVSUP  WSS  tasked  Lockheed  Martin  and  Raytheon  to  establish  processes 
that  reduce  logistics  response  time,  variation  in  the  support  process,  and  the 
total  ownership  cost  for  the  SPY-1  radar  parts  supported  through  the  contracts. 
However,  NAVSUP  WSS  did  not  identify  in  the  contracts,  the  specific  activities  and 
costs  it  required  Lockheed  Martin  and  Raytheon  to  achieve  performance  outcomes 
on,  such  as  the  frequency  of  the  repair  and  parts  replacement.  Without  these 
performance  and  cost  baselines,  NAVSUP  WSS  could  not  identify  deficiencies  with 
the  existing  support  process  or  why  specific  parts  cost  more.  NAVSUP  WSS  did 
not  establish  meaningful  metric  standards  that  provided  Lockheed  Martin  and 
Raytheon  the  incentive  to  reduce  logistics  response  time,  support  process  variation, 
and  total  ownership  costs. 

NAVSUP  WSS  should  review  the  readiness  and  sustainment  performance  history 
and  cost  of  AEGIS  and  SPY-1  radar,  use  that  data  to  identify  the  difference  between 
existing  and  desired  SPY-1  radar  performance  outcomes,  and  develop  metrics  that 
incentivize  the  contractors  to  deliver  the  desired  outcomes. 

Performance  Metrics  Were  Not  Appropriately  Tailored  to 
Desired  Warfighter  Outcomes 

NAVSUP  WSS  did  not  effectively  select  and  refine  the 
contract  metrics  to  achieve  desired  performance 
outcomes  for  the  SPY-1  radar  system.  According 
to  DoD  guidance,25  implementation  of  an 
effective  product  support  strategy  depends  on 
metrics  that  accurately  reflect  the  user's  needs 
and  effectively  measure  the  product  support 
provider's  performance.  Many  PBL  arrangements 
are  executed  at  the  subsystem  or  component  levels. 

Consequently,  system-level  requirements  should  be  broken 
down  into  lower-level  performance  metrics  appropriate  for  the  responsibility 
and  risk  assigned  to  the  product  support  provider.  However,  NAVSUP  WSS 


System-level 
requirements 
should  be  broken 
down  into  lower-level 
performance  metrics 
appropriate  for  the 
responsibility  and  risk 
assigned  to  the  product 
support  provider. 


25  DoD  Product  Support  Manager  Guidebook,  April  2011,  provides  guidance  on  how  to  develop  and  execute  a  product 
support  strategy. 
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bundled  the  SPY-1  radar  sustainment  with  other  AEGIS  subsystem  sustainment 
needs.  Lockheed  Martin's  contract  supports  three  AEGIS  major  subsystems,  and 
Raytheon's  contract  provides  supply  support  for  two  AEGIS  subsystems. 

NAVSUP  WSS  used  consolidated  performance  metrics  to  monitor  Lockheed  Martin 
and  Raytheon's  success  in  filling  part  requests  from  the  fleets  for  all  AEGIS 
subsystems  supported  by  each  respective  contract.  However,  as  Table  3  shows 
when  a  particular  subsystem's  parts  represent  a  low  percent  of  the  total  orders 
filled  through  a  particular  contract,  the  contractor's  poor  performance  can  remain 
undetected  by  a  high-level  metric. 


Table  3.  Contractors’  Success  in  Filling  SPY-1  Parts  Requests  During  First  6-month  Review 


/cm  im 

p  WWW/ 

Lockheed  Martin 

Raytheon 

NAVSUP  WSS 

NAVSUP  WSS 

PBL  Performance 
Metrics 

All  AEGIS  Just  SPY-1 

Subsystems  Radar 

All  AEGIS  Just  SPY-1 

Subsystems  Radar 

Average  Fill  Rate 

■ 

V  W 

Average  Contractor 
Response  Time 

Average  Casualty 
Report  Response  Time 

*  * 

(FOUO) 

Source:  DoD  OIG 


NAVSUP  WSS's  performance  metric  reviews  showed  both  contractors  met  the 
standards  for  all  the  metrics.  However,  our  review  of  the  contractors'  success  in 
filling  just  the  SPY-1  radar  part  orders  found  that  when  the  SPY-1  radar  part  orders 
represented  a  low  percent  of  the  total  orders,  the  contractors’  poor  performance  on 
those  orders  could  remain  undetected  by  the  high-level  metric. 

(FGHG) 

NAVSUP  WSS 

Lockheed  Martin  and  Raytheon  to  fill  at  least  85  percent  of  the  part  orders 
within  the  contracted  delivery  timeframe  to  meet  the  metric  standard  for  the 

average  fill  rate.  f  Yi 
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Tins 

occurred  because  the  SPY-1  radar  orders  represented  only  27.4  percent 

of  the  total  orders  that  Lockheed  Martin  filled  during  the  first 
performance  review  period. 


NAVSUP  WSS  should  design  performance  metrics  that  accurately  reflect  the 
user's  needs  and  effectively  measure  the  product  support  provider's  performance. 
In  addition,  NAVSUP  WSS  should  breakdown  system-level  requirements  into 
lower-level  performance  metrics  that  appropriately  link  contractor  performance 
to  the  accomplishment  of  warfighter  readiness  and  performance  needs. 


Readiness  and  Support  Goals  Achievement  at  Risk 

According  to  the  implementing  directive26  for  DoD’s  Better  Buying  Power  Initiative, 
the  history  of  PBL  contracting  demonstrates  that  DoD  can  achieve  improved 
readiness  at  significant  savings  when  PBL  business  arrangements  are  properly 
structured  and  executed.  Having  appropriate  metrics  aligned  to  warfighter 
requirements  are  critical  to  incentivize  and  motivate  contractors  to  fulfill 
warfighter  outcomes  at  a  reduced  total  ownership  cost.  However,  NAVSUP  WSS 
did  not  take  the  necessary  steps  to  properly  structure  the  performance  metrics  it 
incorporated  in  the  PBL  contracts  used  to  sustain  the  SPY-1  radar. 

NAVSUP  WSS  did  not  consult  with  the  organizations  that  supported  the 
warfighting  combatant  commanders  to  identify  and  translate  their  warfighter 
capability  needs  into  requirements.  Consequently,  the  contract  performance 
metrics  in  both  contracts  did  not: 

•  align  to  warfighter  requirements; 

•  effectively  incentivize  Lockheed  Martin  and  Raytheon  to  make  process 
improvements; 

•  allow  for  decreased  repairs  by  improving  parts  reliability;  or 

•  reduce  total  ownership  costs  associated  with  supporting  the  327  critical 
SPY-1  radar  parts. 

The  operational  availability  of  the  AEGIS  Weapon  System  may  also  be  adversely 
impacted  if  parts  needed  to  maintain  the  SPY-1  radar,  a  critical  AEGIS  subsystem, 
are  not  provided  to  the  warfighters  when  needed  and  cannot  be  obtained  timely 
from  other  sources.  NAVSUP  WSS  needs  to  link  Lockheed  Martin  and  Raytheon's 


26  DoD  memorandum,  "Implementation  Directive  for  Better  Buying  Power  2.0  -  Achieving  Greater  Efficiency  and 
Productivity  in  Defense  Spending,”  April  24,  2013. 


16  |  DODIG-2016-116 


FOR  OFFICIAL  USE  ONLY 


FOR  OFFICIAL  USE  ONLY 


Finding  A 


contract  metrics  to  AEGIS  operational  requirements  and  make  sure  the  metrics 
influence  the  contract  pricing  and  incentives  for  the  PBL  arrangements  to 
be  effective. 

Recommendations,  Management  Comments,  and 
Our  Response 

Recommendation  A.l 

We  recommend  that  the  Commander,  Naval  Supply  Systems  Command,  require  the 
Naval  Supply  Systems  Command  Weapon  Systems  Support  follow  Department  of 
Defense  guidance  when  developing  the  performance  metrics  incorporated  in  future 
performance-based  logistics  contracts  used  to  sustain  the  SPY-1  radar.  Naval 
Supply  Systems  Command  Weapon  Systems  Support  should: 

a.  Consult  with  Advanced  Traceability  and  Control  and  the  operational 
commands  when  reevaluating  the  SPY-1  radar’s  product  support 
strategy  and  designing  the  performance  metrics  included  in  future 
performance-based  logistics  contracts. 

Naval  Supply  Systems  Command  Comments 

The  Commander,  NAVSUP,  agreed  with  the  recommendation.  The  Commander  stated 
that  the  collective  effort  of  both  Navy  governmental  and  industry  stakeholders  was 
essential  to  achieve  and  measure  SPY-1  radar  contract's  performance  outcome.  The 
Commander  stated  NAVSUP  WSS  would  continue  to  consult  all  Navy  stakeholders, 
including  ATAC  and  the  operational  commands,  for  the  SPY-1  contract  renewal 
efforts  to  make  sure  that  NAVSUP  WSS  applies  the  best  performance  outcomes  for 
the  SPY-1  radar.  The  Commander  stated  that  the  contract  renewal  efforts  would 
include  reevaluating  the  performance  metrics  at  various  points  during  the  product 
support  life  cycle. 

Our  Response 

Comments  from  the  Commander,  NAVSUP,  addressed  the  specifics  of  the 
recommendation;  however,  the  Commander,  NAVSUP,  did  not  provide  the  date  that 
NAVSUP  WSS's  actions  would  be  completed.  We  request  the  Commander,  NAVSUP, 
provide  the  date  NAVSUP  WSS's  actions  are  expected  to  be  completed. 
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b.  Establish  formal  support  agreements  with  Advanced  Traceability  and 
Control  and  the  operational  commands  used  to  supply  SPY-1  radar  parts 
to  fleet  customers. 

Naval  Supply  Systems  Command  Comments 

The  Commander,  NAVSUP,  partially  agreed  with  the  recommendation.  The 
Commander  stated  that  coordination  with  ATAC  and  other  stakeholders  was 
necessary  to  properly  define  the  PBL  delivery  metrics  and  other  performance 
outcomes.  However,  the  Commander  emphasized  that  ATAC's  performance  is 
outside  the  scope  and  control  of  the  PBL  provider.  The  Commander  also  stated  it 
would  be  inappropriate  for  NAVSUP  to  enter  into  a  formal  agreement  with  ATAC, 
as  ATAC  was  part  of  NAVSUP  WSS,  which  is  subordinate  to  NAVSUP. 

Our  Response 

Comments  from  the  Commander,  NAVSUP,  did  not  address  the  specifics  of  the 
recommendation.  The  intent  of  our  recommendation  was  not  for  NAVSUP  to  enter 
into  a  formal  agreement  with  ATAC.  Rather,  we  recommended  that  NAVSUP  WSS 
establish  a  formal  agreement  with  ATAC.  NAVSUP  WSS  uses  PBL  contractors  and 
ATAC  to  supply  the  parts  the  fleet  needs  to  sustain  the  SPY-1  radar.  However, 
NAVSUP  Global  Logistic  Support,  another  business  unit  within  NAVSUP,  executes 
the  ATAC  program.  ATAC  picks  up  the  SPY-1  parts  from  the  PBL  contractors  and 
transports  the  parts  to  the  fleet.  While  we  acknowledge  that  ATAC's  performance 
is  outside  the  scope  and  control  of  the  PBL  contractors,  the  contractors  ATAC  uses 
to  transport  parts  do  not  follow  ATAC's  business  rules.  Consequently,  without  a 
formal  support  agreement  requiring  ATAC  to  deliver  the  parts  in  a  set  timeframe, 
we  question  how  NAVSUP  WSS  can  properly  establish  delivery  standards  for  the 
SPY-1  radar  PBL  contracts  and  ensure  the  fleet's  sustainment  needs  are  met.  The 
establishment  of  agreements  with  intragovernmental  entities  involved  in  product 
support  is  consistent  with  DoD  PBL  guidance.  We  request  the  Commander,  NAVSUP, 
provide  comments  to  the  final  report  explaining  how  NAVSUP  will  make  sure  the 
right  delivery  standards  are  incorporated  into  the  follow-on  PBL  contracts  and  the 
fleet's  SPY-1  radar  sustainment  needs  are  met  without  a  formal  agreement  binding 
ATAC  to  deliver  parts  within  set  timeframes. 

c.  Review  the  readiness  and  sustainment  performance  history  and  costs  of 
the  AEGIS  and  SPY-1  radars  and  use  that  data  to  identify  the  difference 
between  existing  and  desired  SPY-1  radar  performance  outcomes,  and 
develop  metrics  that  incentivize  the  contractors  to  deliver  the  desired 
performance  outcomes. 
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d.  Design  performance  metrics  that  accurately  reflect  the  user’s  needs  and 
effectively  measure  the  product  support  provider’s  performance. 

e.  Breakdown  system-level  requirements  into  lower-level  metrics  that 
appropriately  link  contractor  performance  to  the  accomplishment  of 
warfighter  readiness  and  performance  needs. 

Naval  Supply  Systems  Command  Comments 

The  Commander,  NAVSUP,  agreed  with  the  recommendations.  The  Commander 
stated  that  Naval  Sea  Systems  Command  Program  Executive  Office  program 
managers  monitor  the  SPY-1  radar's  operational  availability  and  overall  sustainment 
costs.  The  Commander  stated  that  NAVSUP  WSS  would  obtain  current  performance 
data  and  determine  if  the  PBL  contractors’  performance  is  negatively  impacting 
the  SPY-1  radar's  operational  availability.  The  Commander  also  stated  that 
NAVSUP  WSS  had  developed  performance  metrics  for  the  follow-on  PBL  contracts. 
Specifically,  NAVSUP  WSS  plans  to  include  fill  rate  and  average  contractor  response 
time  metrics  that  hold  both  AEGIS/SPY-1  radar  PBL  providers  to  metric  standards 
consistent  with  Chief  of  Naval  Operations  Instruction  4441. 12D  and  Navy  average 
customer  wait  time  goals.  The  Commander  stated  that  NAVSUP  WSS  would 
work  with  the  Naval  Sea  System  Command  Program  Executive  Office  to  assess 
the  overall  performance  metrics  applicable  to  the  327  SPY-1  radar  unique  parts. 

The  Commander  stated  those  efforts  would  address  if  further  metric  refinement 
and  lower-level  performance  metrics  need  to  be  incorporated  into  the  follow-on 
PBL  contracts,  which  are  estimated  to  be  awarded  by  February  28,  2018. 

Our  Response 

Comments  from  the  Commander,  NAVSUP,  did  not  address  all  the  specifics  of  the 
recommendations.  According  to  the  work  statements  for  the  SPY-1  radar  contracts, 
the  goals  of  the  contracts  were  to  create  a  flexible,  streamlined  process  between 
the  Navy  and  the  PBL  contractors  that  reduced: 

•  administrative  lead  time/cycle  time  (logistics  response  time]; 

•  support  process  variation;  and 

•  the  total  ownership  cost  associated  with  the  327  critical  SPY-1  radar  parts 
supported  in  the  contracts. 
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NAVSUP  WSS's  planned  actions  and  proposed  metrics  focus  only  on  making  sure 
that  the  SPY-1  radar  parts  are  supplied  when  needed.  None  of  the  metrics  that 
NAVSUP  WSS  plans  to  include  in  the  follow-on  PBL  contracts  will  incentivize  the 
contractors  to  accomplish  the  above  goals.  A  properly  designed  PBL  arrangement 
aligns  the  provider's  profit  to  all  the  Government's  goals  through  the  incentives 
included  in  the  contract.  We  request  the  Commander,  NAVSUP,  provide  comments 
to  the  final  report  explaining  what  metrics  NAVSUP  WSS  plans  to  use  to  incentivize 
the  PBL  contractors  to  reduce  logistics  response  time,  support  process  variation, 
and  the  total  ownership  cost  associated  with  the  327  critical  SPY-1  radar  parts 
supported  in  the  contracts. 
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Finding  B 

Inadequate  Performance  Assessments 

NAVSUP  WSS  personnel  did  not  adequately  review  Lockheed  Martin  and 
Raytheon's  success  in  filling  Navy  customers'  SPY-1  radar  part  orders. 

Specifically,  NAVSUP  WSS  personnel: 

•  excluded  orders  from  performance  metric  reviews;  and 

•  incorrectly  calculated  the  length  of  time  contractors  took  to  fill 
part  orders. 

This  occurred  because  NAVSUP  WSS  personnel  did  not  have  written  procedures  to 
evaluate  the  contractors'  performance  toward  meeting  the  contract  performance 
metrics.  As  a  result,  NAVSUP  WSS  paid  the  contractors  $18  million  during 
the  Command's  performance  review  without  deducting  incentive  fees  for  poor 
performance  that  was  not  identified  during  these  reviews.27 


NAVSUP  WSS  Needs  to  Improve  Performance 
Metrics  Reviews 

NAVSUP  WSS  personnel  did  not  adequately  review  Lockheed  Martin  and 
Raytheon's  success  in  filling  Navy  customers'  SPY-1  radar  part  orders.  The 
contractors'  performance  in  filling  SPY-1  part  orders  was  evaluated  against  the 
three  established  metrics,  as  discussed  in  Finding  A,  and  both  contractors  were 
paid  monthly  based  on  their  success  in  meeting  the  standards  for  those  metrics. 

Lockheed  Martin  and  Raytheon  submitted  monthly  reports  on  meeting  the 
performance  metric  standards.  Although  NAVSUP  WSS  tracked  contractor 
performance  monthly,  it  only  evaluated  the  contractors'  success  in  meeting 
their  respective  performance  metric  standards  on  a  semiannual  basis.  During 
the  monthly  reviews,  NAVSUP  WSS  contracting  officials  compared  data  from  the 
Navy  Enterprise  Resource  Planning  system  and  the  Commercial  Asset  Visibility 
database  to  the  contractor's  monthly  reports  of  all  requisitions  submitted. 
NAVSUP  WSS  contracting  officials  used  program  management  personnel  from  the 
NAVSUP  WSS  AEGIS/Ballistic  Missile  Defense  Division  to  evaluate  the  contractors' 
performance  and  determine  whether  an  adjustment  to  the  contractors'  next 
monthly  payment  was  required. 


27  We  audited  NAVSUP  WSS's  two  semiannual  reviews  covering  Lockheed  Martin's  performance  from  August  10,  2012, 
through  August  9,  2013,  and  the  semiannual  reviews  covering  Raytheon's  performance  from  November  28,  2012, 
through  May  27,  2013. 


FOR  OFFICIAL  USE  ONLY 


DODIG-2016-116  |  21 


Finding  B 


FOR  OFFICIAL  USE  ONLY 


According  to  the  contracts,  Lockheed  Martin's  projected  monthly  payment  would 
be  reduced  up  to  3  percent,  while  Raytheon's  reduction  could  be  up  to  9  percent  if 
performance  fell  below  the  metric  standards.  In  addition,  both  contractors  could 
incur  additional  penalties28  if  they  failed  to  deliver  the  parts  within  specified 
timeframes.  Raytheon  was  also  subject  to  a  one-time,  10-percent  penalty  if  it 
failed  to  meet  all  three  performance  metric  standards  for  4  consecutive  months. 

As  of  July  2015,  NAVSUP  WSS  program  management  personnel  had  reviewed 
Lockheed  Martin's  first  year  performance  and  Raytheon's  first  6-month 
performance.  Table  4  shows  the  periods  reviewed,  the  total  number  of  orders 
reviewed,  and  orders  for  SPY-1  radar  parts. 


Table  4.  Completed  NAVSUP  WSS  Semiannual  Performance  Reviews 


lrr\ i  m\ 

\i 

Contractor 

Review  Period 

Total  Orders 
Reviewed 

SPY-1  Radar  Orders 
Reviewed 

Lockheed  Martin 

August  10,  2012 
through 

February  9,  2013 

■ 

■ 

Lockheed  Martin 

February  10,  2013 
through 

August  9,  2013 

■ 

■ 

Raytheon 

November  28,  2012 
through 

May  27,  2013 

■ 

■ 

(FOUO) 

Source:  DoD  OIG 


We  identified  deficiencies  with  the  program  management  personnel’s  conclusions 
related  to  specific  SPY-1  radar  part  orders  filled  by  both  contractors,  during  the 
periods  covered  by  the  reviews.  Specifically,  the  program  management  personnel 
excluded  numerous  SPY-1  radar  part  orders  from  their  review  of  Raytheon's 
performance  and  did  not  consistently  calculate  the  days  both  Lockheed  Martin 
and  Raytheon  took  to  fill  SPY-1  radar  orders.  As  a  result,  NAVSUP  WSS  did  not 
deduct  negative  incentive  fees  from  the  contractors’  payments. 


28  Lockheed  Martin  incurs  a  $1,250  penalty  for  each  routine  order  that  remains  undelivered  for  more  than  agreed 
to  number  of  production  lead-time  days.  Raytheon  incurs  a  $1,000  penalty  for  each  routine  order  that  remains 
undelivered  for  more  than  365  days  and  a  $3,000  penalty  for  each  order  requiring  expedited  delivery  that  remains 
undelivered  for  more  than  9  days. 
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Customer  Orders  Excluded  From  Performance  Metric  Reviews 

NAVSUP  WSS  program  management  personnel  incorrectly  excluded  numerous 
SPY-1  radar  part  orders  from  the  performance  metric 
reviews.  According  to  the  Lockheed  Martin  and 
Raytheon  contracts,  the  average  fill  rate  metric 
is  calculated  from  the  delivery  times  for  all  part 
orders  submitted  during  the  review  period,  and  the 
response  time  metrics  from  all  orders  filled  by  the 
contractors.  The  contracts  addressed  which  part 
orders  should  be  evaluated  as  part  of  the  semiannual 
metric  reviews.  Specifically,  all  recurring  part 
orders29  should  be  filled  by  the  contractors  with  exist 
inventory  and  included  in  all  metric  reviews. 

The  contracts  also  included  provisions  that  allow  the  contractors  to  use  existing 
inventory  to  fill  one-time,  nonrecurring  part  requirements,  such  as  an  initial 
filling  of  or  an  increase  to  a  part's  stock  level.30  Nonrecurring  part  orders  are  only 
included  in  the  reviews  if  the  contractor  concludes  it  has  sufficient  stock  available 
to  fill  those  orders  without  negatively  impacting  its  ability  to  fill  recurring  part 
orders.  If  the  contractor  determines  that  it  cannot  fill  a  nonrecurring  part  order 
without  adversely  impacting  the  filling  of  recurring  orders,  then  NAVSUP  WSS 
will  issue  a  separate  contract  to  fill  the  nonrecurring  order,  and  those  orders  are 
excluded  from  the  metric  reviews. 


flSKlSnBImRSiRlB 


program 
management  personnel 
incorrectly  excluded 
numerous  SPY-1  radar 
part  orders  from  the 
performance  metric 


reviews. 


(FOUO)  However,  NAVSUP  WSS  program  management  personnel  performing  the 
reviews  did  not  consistently  apply  those  procedures  for  Raytheon.  The  Navy 
submitted  nonrecurring  orders  to  the  contractors  as  part  of  a  fleet  Special  Sparing 
Initiative31  to  increase  the  number  of  spare  parts  being  stocked  onboard  ships. 
NAVSUP  WSS  program  management  personnel  did  not  include  in  their  review 
seven  of  the  special  sparing  initiative  orders,  although  Raytheon  used  existing 
contract  inventory  to  fill  the  orders.  In  addition,  the  program  management 
personnel  excluded  47  recurring  orders  for  SPY-1  radar  parts  from  Raytheon's 
performance  metric  review,  because 


29  Recurring  part  orders  represent  a  routine  request  for  an  item  for  use  or  for  stock  replenishment.  The  Navy  informed  the 
contractors  of  the  requirements  for  routine  requests  in  advance. 

30  Nonrecurring  orders  are  submitted  to  Lockheed  Martin  and  Raytheon  without  prior  notification  to  NAVSUP  WSS  and 
were  not  planned  for  when  the  contracts  were  awarded. 

31  The  Special  Sparing  Initiative  resulted  in  unplanned  orders  from  the  fleet  that  the  contractors  did  not  anticipate  and 
include  in  fleet's  forecasted  contract  needs.  For  example,  the  Navy  had  a  sparing  initiative  to  increase  the  number  of 
parts  stocked  onboard  ships  to  support  the  increase  in  the  readiness  goal  of  AEGIS  Weapon  System. 
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(FOUO)  NAVSUP  WSS  program  management  personnel  viewed  special  sparing 
initiative  orders  as  a  unique  requirement  that  should  not  be  included  in  the 
performance  metric  review.  The  personnel  stated  that  they  excluded  the  recurring 
demand  orders  from  the  performance  metrics  because  they  required  the  contractor 
to  fill  the  special  sparing  initiative  orders  before  the  recurring  part  orders.  The 
program  management  personnel  also  considered  it  unfair  to  include  special  sparing 
orders  in  the  performance  metric  review  because  the  orders  were  not  anticipated 
and  the  contractor  received  many  of  these  orders  in  a  short  period  of  time. 

Overall,  program  management  personnel  stated  that  a  Navy's  "programmatic" 
decision  caused  the  contractors 


-  *  .  v.  -  i‘i\  '<  However,  the 

program  management  personnel  could  not  provide  any  documentation  to  support 
this  statement. 


Contractors  Order  Fill  Times  Miscalculated 


NAVSUP  WSS  program  management  officials  incorrectly  calculated  the  length  of 
time  contactors  took  to  fill  part  orders.  The  contracts  required  the  contractors  to 


NAVSUP  WSS 
program 
management  officials 
incorrectly  calculated 
the  length  of  time 
contactors  took  to 
fill  part  orders. 


process  urgent  orders  every  day  and  all  other  orders,  such 
as  routine  orders,  Monday  through  Friday.  According  to 
NAVSUP  WSS  personnel,  they  calculated  the  number  of 
days  the  contractors  took  to  fill  orders  by  subtracting 
the  date  the  orders  were  received  from  the  date  the 
orders  were  shipped.  Consequently,  routine  order 
delivery  time  calculations  included  the  weekends, 
although  the  contracts  state  the  contractor's  delivery 
time  for  routine  orders  only  include  business  days, 
not  weekends. 


(FOUO)  In  addition,  NAVSUP  WSS  program  management  officials  incorrectly 
calculated  the  contractor  delivery  response  time  for  urgent  part  orders.  For 

example, 

Table  5  shows  the  number  of 

SPY-1  part  orders  impacted  by  these  deficiencies. 
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Table  5.  Number  of  Orders  for  SPY-1  Radar  Parts  With  Miscalculated  Fill  Times 


(cm  im 

(l  w  WJ 

Contractor 


Review  Period 


Order  Type 


Total  Orders 
Shipped 


Orders 

Miscalculated 


Lockheed  Martin 

August  10,  2012, 
through 

February  9,  2013 

Routine 

■ 

■ 

Urgent 

■ 

1 

Lockheed  Martin 

February  10,  2013, 
through 

August  9,  2013 

Routine 

■ 

■ 

Urgent 

■ 

1 

Raytheon 

November  28,  2012, 
through 

May  27,  2013 

Routine 

■ 

■ 

Urgent 

■ 

■ 

(FOUO) 

Source:  DoD  OIG 


Additional  Guidance  Needed  for  Performance 
Metric  Reviews 

NAVSUP  WSS  personnel  did  not  have  written  procedures  to  evaluate  the 
contractor's  performance  towards  meeting  the  contract  performance  metrics. 
Specifically,  there  were  no  written  procedures  that  provided  guidance  for  program 
personnel  on  which  orders  to  include  or  exclude  from  performance  metric  reviews 
and  how  to  calculate  the  number  of  days  the  contractors  took  to  fill  the  orders. 
Instead,  we  found  that  during  the  semiannual  performance  reviews,  NAVSUP  WSS 
personnel  made  individualized  determinations  on  hundreds  of  orders  that  the  PBL 
contractors  filled  during  the  period.  Many  orders  had  different  priorities  such  as 
routine,  nonrecurring,  and  expedited  orders,  which  impacted  how  NAVSUP  WSS 
personnel  reviewed  these  orders. 

The  lack  of  written  procedures  caused  inconsistencies  in  which  orders  the  program 
management  officials  determined  should  be  included  in  the  performance  metric 
reviews.  These  inconsistencies  impacted  the  program  officials'  determinations  of 
whether  the  contractors  met  or  did  not  meet  the  performance  metric  standards  on 
specific  orders.  NAVSUP  WSS  should  establish  guidance  and  written  procedures 
that  clearly  describe  the  NAVSUP  WSS  process  for  conducting  semiannual 
performance  metric  reviews  for  the  PBL  contracts. 
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Inconsistencies  May  Have  Adversely  Impacted  Performance  Metric  Review  Outcomes 

After  correcting  NAVSUP  WSS's  review  results  for  improperly  excluding  orders  and  miscalculating  days,  we  determined  that  these 
errors  affected  Raytheon  meeting  the  metric  standards  for  the  SPY-1  radar  orders.  Table  6  compares  the  results  of  NAVSUP  WSS 
with  our  review  to  determine  whether  the  contractors  met  the  metric  standards  for  the  fleet's  SPY-1  part  orders.  It  also  shows 
NAVSUP  WSS  conclusions  for  all  subsystems  part  orders  filled  during  the  review  periods. 


Table  6.  OIG  Assessment  Compared  to  NAVSUP  WSS  Semiannual  Metric  Reviews 


(cr\ i  m\ 

WWW/ 

Lockheed  Martin  Raytheon 


First  Semiannual  Review  Second  Semiannual  Review  First  Semiannual  Review 


Performance 

Metric 

All 

Sub-systems 

NAVSUP WSS 
SPY-1 

DoD  OIG 
Calculations 
SPY-1 

All 

Sub-systems 

NAVSUP  WSS 
SPY-1 

DoD  OIG 
Calculations 
SPY-1 

All 

Sub-systems 

NAVSUP  WSS 
SPY-1 

DoD  OIG 
Calculations 
SPY-1 

Average  Fill  Rate 

Average 
Contractor 
Response  Time 

Average 
Casualty  Report 
Response  Time 

(FOUO) 

Source:  DoD  OIG 


(FOUO)  Our  review  of  Raytheon  results  differed  from  NAVSUP  WSS's  for  the  average  fill  rate  and  average  casualty  report 
response  time  metrics.  NAVSUP  WSS's  inaccurate  reviews  also  failed  to  assess  additional  penalties  to  Raytheon  for  not  filling 
fleet  SPY-1  radar  orders  within  the  timeframe  stated  in  the  contract.  Specifically,  NAVSUP  WSS  program  management  officials 
did  not  deduct  for  orders  that  Raytheon  did  not  fill  within  the  contracted  timeframe.  In  addition,  our  review  concluded 
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(FOUO)  that  NAVSUP  WSS  program  management  officials  failed  to  compensate 
Lockheed  Martin  for  exceeding  performance  expectations.  NAVSUP  WSS  risks 
making  the  same  errors  with  orders  for  parts  associated  with  the  other  subsystems 
supported  through  the  contracts. 

There  is  risk  that  NAVSUP  WSS  made  the  same  errors  with  orders  for  parts 
associated  with  the  other  subsystems  supported  through  the  contracts. 

NAVSUP  WSS  should  perform  additional  reviews  of  the  semiannual  reports  it 
has  completed  to  date,  for  contracts  N00104-12-D-ZD21  and  N00104-13-D-ZD00, 
to  determine  if  there  is  a  change  in  the  amount  of  incentives  the  contractors 
received.  If  NAVSUP  WSS  determines  that  either  contractor  did  not  receive  the 
proper  incentive  payment  or  have  a  penalty  assessed,  NAVSUP  WSS  should  take 
corrective  actions. 

Summary 

NAVSUP  WSS  paid  Lockheed  Martin  and  Raytheon  $18  million  without 
deducting  incentives  fees  for  poor  performance  from  August  10,  2012,  through 
August  9,  2013,  for  the  Lockheed  Martin  PBL  contract  and  November  28,  2012, 
through  May  27,  2013,  for  the  Raytheon  PBL  contract.  NAVSUP  WSS  used  program 
management  personnel  from  the  NAVSUP  WSS  AEGIS/Ballistic  Missile  Defense 
Division  to  evaluate  the  part  orders  received  during  the  performance  review  period 
against  each  metric's  standard.  However,  NAVSUP  WSS  did  not  provide  those 
officials  written  procedures  for  evaluating  the  contractor's  performance  towards 
meeting  the  performance  metrics  in  the  PBL  contracts. 

The  lack  of  written  procedures  resulted  in  the  program  management  officials 
incorrectly  excluding  part  orders  from  the  evaluation  of  Raytheon's  performance. 
The  lack  of  procedures  also  caused  the  program  management  officials  to 
miscalculate  the  length  of  time  both  Lockheed  Martin  and  Raytheon  took  to  make 
each  order  available.  Consequently,  NAVSUP  WSS  may  not  have  deducted  up  to 
5  percent32  of  incentive  fees  when  Raytheon  did  not  meet  the  metric  standards 
during  the  first  half  year  of  the  PBL  contract. 


32  NAVSUP  WSS  may  not  have  deducted  up  to  5  percent  of  incentive  fees  when  Raytheon  did  not  meet  the  metric 
standards  (2  percent  for  the  fill  rate  metric  and  3  percent  for  the  Average  Casualty  Report  Response  Time  metric). 
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Recommendations,  Management  Comments,  and 
Our  Response 

Recommendation  B.l 

We  recommend  that  the  Commander,  Naval  Supply  Systems  Command  require 
the  Naval  Supply  Systems  Command  Weapon  Systems  Support  to: 

a.  Establish  written  procedures  that  clearly  describe  the  process 
for  conducting  semiannual  performance  metric  reviews  for  the 
Performance-Based  Logistics  contracts. 

b.  Perform  additional  reviews  of  the  completed  semiannual  reports  for 
contracts  N00104-12-D-ZD21  and  N00104-13-D-ZD00,  to  determine  if 
there  is  a  change  to  the  amount  of  incentives  the  contractors  received 
and  take  corrective  actions  if  appropriate. 

Naval  Supply  Systems  Command  Comments 

The  Commander,  NAVSUP,  agreed  with  the  recommendations.  The  Commander 
stated  that  NAVSUP  WSS  will  establish  written  procedures  that  clearly  describe 
the  process  for  conducting  semiannual  performance  metric  reviews  for  the 
PBL  contracts.  The  Commander  further  stated  that  NAVSUP  WSS  would  examine 
the  completed  reviews  for  accuracy  and  provide  the  results  to  the  appropriate 
contracting  officer.  The  Commander  stated  that  NAVSUP  WSS  expects  to  complete 
these  actions  by  July  31,  2017. 

Our  Response 

Comments  from  the  Commander,  NAVSUP,  did  not  address  all  the  specifics  of  the 
recommendations.  Although  the  Commander  stated  NAVSUP  WSS  would  provide 
the  results  of  the  reassessment  of  the  completed  performance  metric  reviews  to 
the  appropriate  contracting  officers,  he  did  not  state  what  actions  the  appropriate 
contracting  officials  would  take.  We  request  the  Commander,  NAVSUP,  provide 
comments  to  the  final  report  outlining  what  actions  the  appropriate  contracting 
officer  will  take  after  receiving  the  reassessment  results. 
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Scope  and  Methodology 

We  conducted  this  performance  audit  from  January  2015  through  April  2016 
in  accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives.  We  believe  that  the  evidence  obtained  provides  a 
reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit  objectives. 

We  reviewed  the  Navy's  processes  and  procedures  used  to  determine  whether 
NAVSUP  WSS  developed  and  incorporated  appropriate  performance  metrics  to 
sustain  the  SPY-1  radars  and  adequately  assessed  how  effectively  Lockheed  Martin 
and  Raytheon  filled  Navy  customers'  orders  for  SPY-1  radar  parts.  We  obtained 
a  list  of  all  systems  on  Arleigh  Burke-class  destroyers  and  the  cost  per  system. 

From  this  list,  we  selected  the  AN/SPY-ID  and  D(V]  Phased  Array  Radar  systems, 
as  they  represented  the  two  highest  cost  systems.  We  then  requested  the 
two  PBL  contracts  and  related  contracting  files  to  assess  how  NAVSUP  WSS 
established  the  performance  metrics  included  in  the  contract  used  to  sustain 
the  SPY-1  radars.  In  addition,  we  requested  NAVSUP  WSS  semiannual  reviews 
of  the  contractors'  performance  in  filling  customer  orders  for  the  SPY-1  radar 
system  against  the  metric  standards  established  by  NAVSUP  WSS.  Specifically, 
the  semiannual  reviews  covered  Raytheon's  performance  during  the  period  from 
November  28,  2012,  through  May  27,  2013,  and  Lockheed  Martin's  performance 
from  August  10,  2012,  through  August  9,  2013. 

We  visited  and  interviewed  personnel  from  Weapon  Systems  Support  and 
Business  Systems  Center  at  NAVSUP,  Mechanicsburg,  Pennsylvania.  We  discussed 
with  Navy  officials'  their  process  and  procedures  for  incorporating  appropriate 
performance  metrics  that  effectively  incentivize  support  providers  efforts  in 
achieving  warfighters  desired  outcomes  and  assessing  semiannual  performance 
reviews.  The  contractors  reviewed  relevant  portions  of  the  draft  report  and  one 
contractor  provided  comments,  which  we  considered  in  preparing  the  final  report. 

We  reviewed  the  following  DoD  guidance  applicable  to  the  performance  of 
PBL  contracts. 

•  ASD(L&MR],  "The  DoD  Product  Support  Manager  Guidebook,"  April  2011 
(2011  Release] 

•  ASD(L&MR]  memorandum,  "Performance  Based  Logistics  Comprehensive 
Guidance,"  November  22,  2013 


FOR  OFFICIAL  USE  ONLY 


DODIG-2016-116  |  29 


Appendixes 


FOR  OFFICIAL  USE  ONLY 


•  Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition], 
"Performance  Based  Logistics  (PBL)  Guidance  Document,"  January  27,  2003 

•  Naval  Sea  Systems  Command,  "A  Program  Manager  Guide  to  the  Application 
of  Performance  Based  Logistics  [PBL],"  Revision  A,  October  8,  2008 

Use  of  Computer-Processed  Data 

We  relied  on  the  computer-processed  data  obtained  from  two  systems:  the  Navy 
Enterprise  Resource  Planning  (ERP]  and  Birdtrack.  The  data  in  the  semiannual 
reviews  that  we  used  to  perform  our  analysis  was  retrieved  from  the  ERP 
system.  The  Navy  ERP  provides  consolidated  financial  management,  workforce 
management,  wholesale  and  retail  supply,  plant  maintenance,  and  project 
management  functions  within  the  Navy.  The  supply  support  function  of  ERP 
involves  processing  requisitions,  to  include  requisitions  from  the  fleet  systems 
through  the  Defense  Automated  Addressing  System.  The  Defense  Automated 
Addressing  System  processes  requisitions  using  a  standard  format  called  Military 
Standard  Requisitioning  and  Issue  Procedures  that  uses  codes  to  assist  the  supply 
sources  in  making  supply  decisions  such  as  the  urgency  of  a  requisition.  After  the 
ERP  processes  the  requisition,  ERP  sends  the  requisition  through  the  Commercial 
Asset  Visibility  system  to  the  contractor  to  fill  the  requisition.  Some  of  the  data 
from  Military  Standard  Requisitioning  and  Issue  Procedures  is  captured  in  ERP 
and  included  in  the  semiannual  reviews  are  the  requisition  numbers,  order  dates, 
shipped  dates,  and  received  dates. 

We  traced  a  statistical  sample,  with  assistance  from  the  DoD  OIG,  Quantitative 
Methods  Division  (QMD]  to  the  ERP  requisition  source  document  verifying  the 
reliability  of  the  semiannual  reviews  data.  The  ERP  requisitions  population 
for  Lockheed  Martin  consisted  of  466  requisitions  and  Raytheon  consisted  of 
419  requisitions.  QMD  randomized  the  requisitions  populations.  We  sorted  each 
population  by  the  random  number  column  and  selected  the  first  44  requisitions  for 
Lockheed  Martin  and  the  first  43  for  Raytheon  as  the  basis  for  our  test.  Based  on 
finding  a  zero  error  rate  for  each  sample,  we  concluded  with  90-percent  confidence 
that  the  error  rate  would  be  less  than  five  percent. 

Birdtrack  is  an  automated  system  that  provides  the  Navy  with  parts  tracking 
and  analysis  capabilities  to  speed  up  the  flow  of  replacement  parts  to  ships  and 
forward-deployed  activities.  NAVSUP  personnel  provided  a  list  of  requisitions  by 
part  for  each  destroyer  from  the  Birdtrack  system  that  customers  submitted  to 
NAVSUP  and  the  time  it  took  to  obtain  that  part.  We  used  the  data  to  determine 
the  customer  required  delivery  dates  and  whether  the  parts  were  delivered  in  the 
specified  timeframes.  We  traced  a  statistical  random  sample,  with  assistance  from 
the  QMD,  to  the  Birdtrack  requisition  source  document  to  verify  the  reliability 
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of  the  requisitions  data.  The  Birdtrack  requisitions  population  consisted  of 
2,601  requisitions.  QMD  randomized  the  requisitions  population  and  sorted 
them  in  random  order.  We  selected  the  first  78  requisitions  as  the  basis  for  our 
test.  Based  on  a  zero  but  not  more  than  one  error,  we  could  conclude  that  with 
90-percent  confidence  the  error  rate  in  the  population  would  be  less  than  or  equal 
to  5  percent. 

Based  on  our  reviews,  we  concluded  that  the  data  used  from  the  ERP  and 
Birdtrack  were  sufficiently  reliable  for  the  purposes  of  this  report.  We  did  not 
rely  on  data  from  Commercial  Asset  Visibility,  Defense  Automated  Addressing 
System,  or  contractor  systems  data;  therefore,  we  did  not  test  the  reliability  of 
data  from  these  systems. 

Use  of  Technical  Assistance 

During  the  audit,  we  relied  on  technical  assistance  provided  by  QMD  at  the 
DoD  OIG.  A  QMD  analyst  developed  a  random  sampling  plan  and  provided 
assistance  with  selecting  the  statistical  sample  of  Lockheed  Martin  and 
Raytheon  customer  requisitions  used  to  perform  data  testing  verification 
of  computer-processed  data. 

Prior  Coverage 

During  the  last  5  years,  the  Department  of  Defense  Inspector  General  (DoD  IG] 
issued  seven  reports  discussing  the  management  of  Navy's  spare-part  inventory 
and  the  inventory  management  practices  of  the  Military  Departments.  Unrestricted 
DoD  IG  reports  can  be  accessed  at  http://www.dodig.mil/pubs/index.cfm. 

DoD  IG 

DODIG-2016-011,  "The  Navy  Needs  to  Improve  the  Management  of  Parts  Required 
to  Sustain  the  AN/SPY-1  Phased  Array  Radar  System,"  November  6,  2015 

DODIG-2015-053,  "Naval  Supply  Systems  Command  Needs  to  Improve 
Cost  Effectiveness  of  Purchases  for  the  Phalanx  Close-In  Weapon  System,” 

December  19,  2014 

DODIG-2015-052,  "Air  Force  Life  Cycle  Management  Center's  Management  of 
F119  Engine  Spare  Parts  Needs  Improvement,"  December  19,  2014 

DODIG-2014-119,  "Excess  Inventory  Acquired  on  Performance-Based  Logistics 
Contracts  to  Sustain  the  Air  Force's  C-130J  Aircraft,"  September  22,  2014 
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DODIG-2014-064,  "Improved  Management  Needed  for  the  F/A-18  Engine 
Performance-Based  Logistics  Contracts,"  April  25,  2014 

D-2012-102,  "Better  Cost-Control  Measures  Are  Needed  on  the  Army's 
Cost-Reimbursable  Services  Contract  for  Logistics  Support  of  Stryker  Vehicles," 
June  18,  2012 

D-2011-061,  "Excess  Inventory  and  Contract  Pricing  Problems  Jeopardize  the  Army 
Contract  with  Boeing  to  Support  the  Corpus  Christi  Army  Depot,"  May  3,  2011 
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Appendix  B 

Product  Support  Strategy  Process  Model 

The  product  support  strategy  model  describes  a  12-step  process  to  develop  and 
implement  product  support  strategies.  This  model  represents  the  major  activities 
required  to  implement,  manage,  evaluate,  and  refine  product  support  over  the  life 
cycle.  It  is  not  a  one-time  process  but  rather  a  continuing,  iterative  process  in 
which  the  sustainment  of  a  system  is  adapted  and  evolved  to  optimally  support  the 
needs  and  requirements  of  the  warfighter  in  an  effective  and  affordable  manner. 
Program  officials  should  perform  the  following  12  steps  when  developing  or 
revising  its  product  support  strategy: 

1.  Integrate  Warfighter  Requirements  and  Support.  Translate  system 
operational  requirements  into  the  necessary  sustainment  strategy  for 
effectively  delivering  those  requirements.  The  objective  of  product 
support  is  to  develop,  enable,  and  execute  a  sustainment  strategy 
that  will  deliver  optimum  operational  readiness  to  the  warfighter, 
consistent  with  warfighter  requirements,  at  an  affordable,  best  value 
cost.  Warfighter  requirements  are  expressed  in  operational  terms. 

Those  requirements  must  be  interpreted  and  translated  as  needed  into 
sustainment  objectives  that  will  drive  the  achievement  of  those  outcomes. 

2.  Form  the  Product  Support  Management  Integrated  Product  Support 
Team  (IPT).  The  IPT  consists  of,  but  is  not  limited  to:  logistics  (supply 
and  transportation  staff],  requirements,  operational  mission  planning, 
financial,  contracts,  legal,  and  integrated  product  support  elements 
functional  subject  matter  experts. 

3.  Baseline  the  System.  Collect  data  that  will  be  needed  to  assess  and 
analyze  support  decisions,  including  inputs  from  various  analyses.  This 
data  includes,  but  is  not  limited  to,  the  level  of  repair  analysis,  reliability 
centered  maintenance  analysis,  as  well  as  reliability,  availability,  and 
maintainability  and  life-cycle  cost  analysis.  Defining  and  documenting 
the  system  baseline  involves  identifying  the  historical  readiness  rates 
and  operations  and  support  costs. 

4.  Identify/Refine  Performance  Outcomes.  Develop  a  process  to  identify 
critical  product  support  outcomes  and  to  measure  success.  Identify  the 
critical  behaviors  that  must  be  influenced  by  metrics  to  achieve  product 
support  strategy  outcomes.  The  starting  points  for  metrics  identification 
are  warfighter  outcomes  and  Office  of  the  Secretary  of  Defense's  specified 
top-level  weapon  system  metrics.  Each  product  support  strategy,  as 

it  evolves,  must  be  tailored  consistent  with  the  maturity  of  data  and 
existence  of  in-place  support  infrastructure  and  capabilities.  The 
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metrics  defined  as  accountable  outcomes  must  be  tailored  accordingly 
with  an  objective  to  maintain  a  close  correlation  with,  and  enable  the 
achievement  of,  the  warfighter  and  Office  of  the  Secretary  of  Defense's 
top-level  outcomes. 

5.  Business  Case  Analysis.  Assess  the  cost,  competencies,  capabilities,  and 
process  efficiencies  to  identify  the  optimum  best  value  product  support 
solution.  The  goal  of  the  Product  Support  or  Sustainment  Business  Case 
Analysis  (BCA]  is  to  identify  the  product  support  strategy  that  achieves 
the  optimal  balance  between  warfighter  capabilities  and  affordability. 

The  BCA  should  be  a  full,  fair,  and  accurate  comparison  when  evaluating 
multiple  alternatives.  A  BCA  is  used  for  major  life-cycle,  sustainment,  and 
other  product-support  decisions,  especially  those  that  result  in  new  or 
changed  resource  requirements. 

6.  Product  Support  Strategy  Value  Analysis.  Is  a  best  value  analysis  to 
optimize  long-term,  life-cycle  costs  and  benefits.  This  analysis  includes, 
but  is  not  limited  to:  the  supply  chain  management  strategy;  strategies 
for  continuous  modernization  and  improvement  of  system  reliability; 
availability  and  maintainability;  proactively  addressing  obsolescence; 
diminishing  manufacturing  sources  and  material  shortages;  corrosion 
issues;  a  life-cycle  cost  control;  and  risk  mitigation. 

7.  Determine  Support  Methods(s).  Determine  whether  support  will  be 
acquired  from  product  support  providers  (for  example,  contractors, 
and/or  government]  using  an  outcome-based  or  transactional-based 
acquisition  method.  Decisionfs]  are  validated  or  made  using  a  best  value 
analysis  consistent  with  the  BCA.  There  are  ultimately  only  two  options 
available  to  them  with  some  variations  between  these  two  options.  They 
can  either  acquire  the  discrete  goods  and  services  necessary  to  enable 
the  required  warfighter  outcomes,  or  they  can  acquire  the  outcomes 
themselves.  The  former  is  the  transactional  support  model,  and  the 
latter  is  the  performance-based  (or  outcome-based]  model.  DoD  policy 
and  guidance  specifies  a  preference  for  the  performance-based  model 
wherever  possible. 

8.  Designate  Product  Support  Integrator(s)  (PSIs).  For  outcome-based 
support,  identify  the  Product  Support  Integrators]  who  will  be 
delegated  the  responsibility  to  integrate  support  providers  to  deliver 
the  specified  outcomes  assigned  consistent  with  the  scope  of  their 
delegated  responsibility.  Decision(s]  are  validated  or  made  using  a 
best  value  analysis  consistent  with  the  BCA.  The  program  manager's 
responsibilities  for  oversight  and  management  of  the  product  support 
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function  are  typically  delegated  to  the  Product  Support  Manager,  who 
leads  the  development  and  implementation  of  the  product  support 
strategies  and  ensures  achievement  of  desired  support  outcomes  during 
sustainment.  As  with  the  product  support  strategy  and  the  arrangement 
with  the  warfighter,  the  PSI  function  is  a  key  component  of  the  product 
support  strategy  documented  in  the  acquisition  strategy  and  the  life-cycle 
sustainment  plan.  While  product  support  execution  is  accomplished  by 
numerous  organizational  entities  (also  called  Product  Support  Providers], 
the  PSI  is  the  single  point  of  accountability  for  integrating  all  sources  of 
support  necessary  to  meet  the  agreed  to  support/performance  metrics. 
Anyone  who  provides  products  or  services  in  the  sustainment  of  an 
acquisition  system  is  a  Product  Support  Provider.  The  primary  role  of 
the  Product  Support  Integrator  is  to  integrate  the  activities  of  the  various 
Product  Support  Providers. 

9.  Designate  Product  Support  Provider(s).  Use  the  BCA  value  analysis,  as 
well  as  PSI  discretionary  decisions  for  lower  tiered  supplier  support,  to 
select  the  best  mix  and  blend  of  sources  to  perform  the  product  support 
functions.  Decision^]  are  validated  or  made  using  a  best  value  analysis 
consistent  with  the  BCA.  A  primary  objective  of  the  BCA  process  is  to 
determine,  for  the  individual  Integrated  Product  Support  elements  and, 
in  aggregate,  the  objective  system,  the  optimum  sources  of  support 
depending  on  capabilities,  competencies,  best  value,  and  the  qualitative 
efficiency  and  effectiveness  of  support.  For  each  of  the  Integrated  Product 
Support  elements  there  will  be  logical  candidates,  both  public  and  private, 
to  accomplish  the  required  product  support. 

10.  Identify/Refine  Financial  Enablers.  Identify  the  range,  types,  and  amount 
of  funding  required  to  accomplish  the  required  support  consistent  with 
the  terms,  conditions,  and  objectives  of  the  Product  Support  Agreements. 

11.  Establish/Refine  Product  Support  Agreements.  Document  the 
implementing  support  arrangements  (such  as,  a  contract,  memorandum 
of  agreement,  and  memorandum  of  understanding]  that  assign  and 
delineate  the  roles,  responsibilities,  resourcing,  and  reciprocal  aspects 

of  product  support  business  relationships.  The  Product  Support  Manager 
should  ensure  the  Product  Support  Agreements  are  in  place  to  document 
and  define  each  relationship  that  is  part  of  the  execution  of  the  product 
support  strategy.  These  Product  Support  Arrangements  serve  to  formalize 
the  roles,  responsibilities,  relationships,  and  commitments  of  the  active 
participants  in  the  product  support  strategy. 
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12.  Implement  and  Oversight.  Implement  and  manage  the  product 

support,  including  documenting  updates  to  the  life-cycle  sustainment 
plan,  conducting  and  implementing  recommendations  from  logistics 
assessments,  and  maturing  the  sustainment  maturity  level.  This  includes 
the  continuous,  ongoing  assessment  of  Product  Support  effectiveness 
through  using  the  established  governance  mechanisms  driving  decisions 
and  actions  to  review,  modify,  revise,  or  evolve  product  support  strategies 
and  business  arrangements.  The  Product  Support  Manager's  oversight 
role  includes  developing  the  performance  assessment  plan,  monitoring 
performance,  and  revising  the  life-cycle  sustainment  plan  and  Product 
Support  Package  as  needed.  The  Program  Manager  also  acts  as  the  agent 
for  the  warfighter,  certifying  PSI  performance  and  approving  incentive 
allocations.  The  Product  Support  Manager  should  take  a  hands-on 
approach  and  not  assume  that  the  Product  Support  Agreement  will  be 
self-regulating. 
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Urgent  Orders  Filled  and  Not  Delivered  by  Customers' 
Required  Delivery  Date 


Table  7.  Urgent  Orders  Filled  by  Lockheed  Martin  and  Not  Delivered  by  Customers’ 
Required  Delivery  Date 
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Table  7.  Urgent  Orders  Filled  by  Lockheed  Martin  and  Not  Delivered  by  Customers’ 
Required  Delivery  Date  (cont'd) 
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Table  7.  Urgent  Orders  Filled  by  Lockheed  Martin  and  Not  Delivered  by  Customers’ 
Required  Delivery  Date  (cont'd) 
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Table  7.  Urgent  Orders  Filled  by  Lockheed  Martin  and  Not  Delivered  by  Customers’ 
Required  Delivery  Date  (cont'd) 
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Table  8.  Urgent  Orders  Filled  by  Raytheon  and  Not  Delivered  by  Customers’  Required 
Delivery  Date 
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Table  8.  Urgent  Orders  Filled  by  Raytheon  and  Not  Delivered  by  Customers’  Required 
Delivery  Date  (cont'd) 
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Table  8.  Urgent  Orders  Filled  by  Raytheon  and  Not  Delivered  by  Customers’  Required 
Delivery  Date  (cont'd) 
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Management  Comments 


Management  Comments 

Naval  Supply  Systems  Command 


DEPARTMENT  OF  THE  NAVY 

NAVAL  SUPPLY  SYSTEMS  COMMAND 
5450  CARLISLE  PIKE 
PO  BOX  2050 

MECHANICSBURG  PA  17055-0791  in  reply  refer  To 

7510 

Ser  NOIG/024 

JUN  1  4.2016 

From:  Commander,  Naval  Supply  Systems  Command 

To:  Department  of  Defense  Inspector  General  -  Deputy,  Assistant  Inspector  General 

Acquisition  and  Sustainment  Management 

Subj:  NAVSUP  COMMENTS  ON  DODIG  DRAFT  AUDIT  REPORT,  “NAVY  NEEDS  TO 
ESTABLISH  EFFECTIVE  METRICS  TO  ACHIEVE  DESIRED  OUTCOMES  FOR 
THE  SPY-1  RADAR  SUSTAINMENT” 

Ref:  (a)  DoDIG  Draft  Audit  Report,  “Navy  Needs  to  Establish  Effective  Metrics  to  Achieve 
Desired  Outcomes  for  the  SPY-1  Radar  Sustainment” 

Enel:  (1)  NAVSUP  Comments  on  DoDIG  Draft  Audit  Report,  “Navy  Needs  to  Establish 

Effective  Metrics  to  Achieve  Desired  Outcomes  for  the  SPY-1  Radar  Sustainment” 


1.  Enclosure  (1)  is  provided  in  response  to  reference  (a). 


Copy  to: 
NAVINSGEN 
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Naval  Supply  Systems  Command  (cont'd) 


NAVSUP  COMMENTS  ON  D0DI6  DRAFT  AUDIT  REPORT,  "NAVY  NEEDS  TO 
ESTABLISH  EFFECTIVE  METRICS  TO  ACHIEVE  DESIRED  OUTCOMES  FOR  THE 
SPY-1  RADAR  SUSTAINMENT" 

DODIG  AUDIT  SUMMARY  OF  FINDINGS: 

Naval  Supply  Systems  Command  (NAVSUP)  Weapon  Systems  Support 
(WSS)  did  not  develop  and  incorporate  appropriate  performance 
metrics  into  the  performance-based  logistics  contracts  used  to 
sustain  SPY-1  radars.  Specifically,  the  metrics  did  not 
effectively  incentivize  Lockheed  Martin  and  Raytheon  to  achieve 
Navy  warfighter  requirements  or  reduce  the  total  ownership  cost 
associated  with  the  327  critical  SPY-1  radar  parts  supported  by 
the  contracts.  This  condition  occurred  because  NAVSUP  WSS 
personnel  did  not  follow  DoD  guidance  when  developing  the 
performance  metrics.  As  a  result,  supply  support  and  cost 
reduction  objectives  for  SPY-1  radar  parts  were  not  met.  In 
addition,  operational  availability  of  the  AEGIS  Weapon  System 
could  be  adversely  impacted  if  parts  needed  to  maintain  the  SPY- 
1  radars  are  not  transported  to  the  warfighters  when  needed. 

Furthermore,  NAVSUP  WSS  personnel  did  not  adequately  assess  the 
contractors'  performance  against  established  metrics.  This 
occurred  because  NAVSUP  WSS  did  not  have  written  procedures  to 
evaluate  contractors'  performance  toward  meeting  the  contract 
metrics . 


RECOMMENDATIONS :  We  recommend  that  the  Commander,  Naval  Supply 

Systems  Command  (NAVSUP) ,  require  NAVSUP  Weapon  Systems  Support 
(WSS)  follow  Department  of  Defense  guidance  when  developing  the 
performance  metrics  incorporated  in  future  performance-based 
logistics  contracts  used  to  sustain  the  SPY-1  radar.  NAVSUP  WSS 
should: 


1 

Enclosure  (1) 
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Management  Comments 


Naval  Supply  Systems  Command  (cont'd) 


Recommendation  A. 1 . a :  Consult  with  Advanced  Traceability  and 
Control  program  and  the  operational  commands  when  reevaluating 
the  SPY-1  radar's  product  support  strategy  and  designing  the 
performance  metrics  included  in  future  performance-based 
logistics  contracts. 

NAVSUP  Comments :  Concur.  NAVSUP  WSS  will  continue  to  consult 
with  all  Navy  stakeholders,  including  NAVSUP  WSS  Advanced 
Traceability  and  Control  (ATAC)  program  and  operational 
commands,  on  our  performance-based  logistics  ( PBL)  renewal 
efforts  to  ensure  we  are  applying  the  best  performance  outcomes 
for  the  SPY-1  Radar.  This  includes  reevaluation  of  metrics  at 
appropriate  points  in  the  product  support  life  cycle.  The 
collective  effort  of  both  Navy  governmental  and  industry 
stakeholders  is  essential  to  achieve  and  measure  the  performance 
outcome  included  in  the  scope  of  the  two  SPY-1  Radar  contracts. 
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Naval  Supply  Systems  Command  (cont'd) 


Recommendation  A.l.b:  Establish  formal  support  agreements  with 
Advanced  Traceability  and  Control  and  the  operational  commands 
used  to  supply  SPY-1  radar  parts  to  fleet  customers. 

NAVSUP  Comments:  Partially  Concur.  Agree  that  coordination 
with  NAVSUP  WSS  ATAC  program  and  other  stakeholders  is  necessary 
to  properly  define  PBL  delivery  metrics  and  other  performance 
outcomes.  ATAC  input  and  participation  is  important  on  any  PBL 
Integrated  Product  Team.  NAVSUP  WSS  will  continue  to  work  with 
our  ATAC  program  and  other  Transportation  Subject  Matter  Experts 
to  ensure  the  Navy  is  providing  the  most  agile  response  to  the 
customer  from  the  supply  system.  However,  it  should  be 
clarified  that  ATAC  program  performance  is  outside  the  scope  and 
control  of  the  PBL  provider  and  is  not  part  of  the  PBL 
contractual  arrangement  with  industry.  In  addition,  we  feel  it 
would  not  be  appropriate  for  Naval  Supply  Systems  Command  to 
enter  into  a  "formal  support  agreement"  with  ATAC,  as  ATAC  is  a 
part  of  NAVSUP  WSS  which  is  subordinate  to  Naval  Supply  Systems 
Command. 
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Naval  Supply  Systems  Command  (cont'd) 


Recommendation  A . 1 ■ c :  Review  the  readiness  and  sustainment 
performance  history  and  costs  of  the  AEGIS  and  SPY-1  radars  and 
use  that  data  to  identify  the  difference  between  existing  and 
desired  SPY-1  radar  performance  outcomes,  and  develop  metrics 
that  incentivize  the  contractors  to  deliver  the  desired 
performance  outcomes. 

NAVSUP  Comments:  Concur.  The  current  Sustainment,  Readiness, 
and  Acquisition  Program  Managers  from  the  Naval  Sea  Systems 
Command  (NAVSEA)  Program  Executive  Office  (PEO)  Integrated 
Warfare  Systems  (IWS)  are  responsible  for  monitoring  the  weapon 
systems  Operational  Availability  (Ao) ,  and  overall  sustainment 
costs  of  the  SPY-1  Radar.  Negative  performance  and  cost  drivers 
for  the  SPY-1  Radar  are  identified  and  reviewed  at  the  quarterly 
AEGIS/SPY-1  Radar  Program  Review  Board  and  also  reviewed  at  a 
monthly  AEGIS  Fleet  Issues  Working  Group.  These  two  groups  work 
to  identify  any  support  gaps,  recommend  any  improvements  to  any 
of  the  Integrated  Product  Support  elements  (including  supply 
support),  and  identify  actions  that  will  lead  to  improved  Ao. 
NAVSUP  WSS  will  obtain  current  Ao  performance  data  related  to 
the  SPY-1  Radar  from  the  NAVSEA  PEO  IWS  1.0  Readiness  team  to 
see  if  there  has  been  any  negative  impact  to  current  Ao 
performance  of  the  SPY-1  Radar  based  on  the  PBL  provider's 
current  performance.  The  AEGIS/SPY-1  PBL  Renewal  Government 
Integrated  Product  Team  will  address  if  the  proposed  performance 
metrics  for  the  follow-on  PBLs  will  need  to  be  further  refined. 
Any  changes  to  the  metrics  will  be  incorporated  into  the  follow- 
on  PBL  which  are  expected  to  be  awarded  by  February  of  2018. 

ECD :  28  February  2018. 
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Naval  Supply  Systems  Command  (cont'd) 


Recommendation  A. 1 . d:  Design  performance  metrics  that 
accurately  reflect  the  user's  needs  and  effectively  measure  the 
product  support  provider's  performance. 

NAVSUP  Comments :  Concur.  NAVSUP  WSS  has  developed  the 
performance  metrics  for  the  follow-on  AEGIS/SPY-1  PBLs .  NAVSUP 
WSS  will  hold  both  AEGIS/SPY-1  Radar  PBL  providers  to  an  85% 
fill  rate  per  the  OPNAV  444 1.12D,  12  April  2012  instruction. 
OPNAVINST  4441. 12D  cites  the  supply  availability  goals  which 
represent  how  often  material  is  on  hand  and  available  to  be 
issued  when  a  requisition  is  received.  Wholesale  level  systems 
that  do  not  use  Multi-Indenture  Multi-Echelon  sparing  models, 
such  as  the  AEGIS/SPY-1  Radar,  are  required  to  meet  a  goal  of 
85%.  That  is,  85%  of  requisitions  will  have  to  be  filled  in 
compliance  with  the  established  performance  delivery 
requirements  of  24  hours  for  Casualty  Report  (CASREP) 
requisitions,  two  days  for  Issue  Priority  Group  (IPG)  I 
requisitions,  four  days  for  IPG  II  requisitions,  and  six  days 
for  IPG  III  requisitions.  In  addition  to  filling  85%  of  the 
requisitions  in  accordance  with  the  delivery  requirement,  the 
PBL  Providers  will  also  maintain  a  specific  Average  Contractor 
Response  Time  ( ACRT )  for  both  CASREP  (1  day)  and  non-CASREP  (7.5 
days)  requisitions.  These  delivery  requirements  were  also 
developed  in  accordance  with  OPNAVINST  4441. 12D  and  the  Average 
Customer  Wait  Time  (ACWT)  goals.  ACWT  is  a  comprehensive 
measure  of  the  time  elapsed  between  the  customer  requirement 
submission  time  and  the  receipt  time  by  the  customer.  ACWT  is  a 
collective  indicator  of  the  Logistics  Response  Time  (LRT)  and 
includes  processing  time  onboard  the  ship.  The  LRT  is  made  up 
of  the  requisition  submission  time,  the  Inventory  Control  Point 
processing  time,  depot  processing  time,  transportation  time,  and 
receipt  take-up  time  and  is  limited  to  off-ship  processing.  The 
delivery  requirements  required  under  the  AEGIS/SPY-1  PBLs  will 
aim  to  decrease  the  depot  processing  time  -  that  is,  the  ACRT  - 
in  order  to  meet  the  total  ACWT  objective  and  the  Navy's  LRT 
goal  of  23  days.  NAVSUP  WSS  will  obtain  current  Operational 
Availability  (Ao)  performance  data  related  to  the  SPY-1  Radar 
from  the  NAVSEA  Program  Executive  Office  Integrated  Warfare 
Systems  1.0  Readiness  team  to  see  if  there  has  been  any  negative 
impact  to  current  Ao  performance  of  the  SPY-1  Radar  based  on  the 
PBL  provider's  current  performance.  The  AEGIS/SPY-1  PBL  Renewal 
Government  Integrated  Product  Team  will  address  if  the  proposed 
performance  metrics  for  the  follow-on  PBLs  will  need  to  be 
further  refined.  Any  changes  to  the  metrics  will  be 
incorporated  into  the  follow-on  PBLs  which  are  expected  to  be 
awarded  by  February  of  2018.  ECD:  28  February  2018. 
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Management  Comments 


Naval  Supply  Systems  Command  (cont'd) 


Recommendation  A . 1 . e :  Breakdown  system-level  requirements  into 
lower-level  metrics  that  appropriately  link  contractor 
performance  to  the  accomplishment  of  warfighter  readiness  and 
performance  needs. 

NAVSUP  Comments:  Concur.  NAVSUP  WSS  will  work  with  the  NAVSEA 
Program  Executive  Office  (PEO)  Integrated  Warfare  Systems  to 
review  and  assess  the  overall  performance  metrics  of  the  327 
SPY-1  unique  items  to  see  if  lower-level  performance  metrics 
will  need  to  be  developed  for  the  follow-on  contracts.  NAVSUP 
WSS  will  obtain  current  Operational  Availability  (Ao) 
performance  data  related  to  the  SPY-1  Radar  from  the  NAVSEA  PEO 
Integrated  Warfare  Systems  1.0  Readiness  team  to  see  if  there 
has  been  any  negative  impact  to  current  Ao  performance  of  the 
SPY-1  Radar.  As  a  result  of  this  analysis,  the  PBL  Renewal 
Government  IPT  will  address  if  lower  level-level  performance 
metrics  will  need  to  be  implemented  into  the  follow-on  PBL 
contracts.  Any  changes  to  the  metrics  will  be  incorporated  into 
the  follow-on  PBLs  which  are  expected  to  be  awarded  by  February 
of  2018.  ECD :  28  February  2018. 
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Naval  Supply  Systems  Command  (cont'd) 


Recommendation  B . 1 . a :  Establish  written  procedures  that  clearly 
describe  the  process  for  conducting  semiannual  performance 
metric  reviews  for  the  Performance-Based  Logistics  contracts. 

NAVSUP  Comments:  Concur.  NAVSUP  WSS  will  work  to  establish  a 
standard  operating  procedure  for  conducting  their  Maritime  PBL 
Semi-Annual  Performance  Reviews.  ECD:  31  July  2017. 
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Management  Comments 


Naval  Supply  Systems  Command  (cont'd) 


Recommendation  B . 1 . b :  Perform  additional  reviews  of  the 
completed  semiannual  reports  for  contracts  N00104-12-D-ZD21  and 
N00104-13-D-ZD00,  to  determine  if  there  is  a  change  to  the 
amount  of  incentives  the  contractors  received  and  take 
corrective  actions  if  appropriate. 

NAVSUP  Comments :  Concur.  NAVSUP  WSS  will  review  the  completed 
performance  reviews  for  accuracy.  Any  changes  to  those 
reassessed  performance  reviews  will  be  provided  to  the 
appropriate  NAVSUP  WSS  Contracting  Officer.  ECD :  31  July  2017. 
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Acronyms  and  Abbreviations 


ASD(L&MR) 

Assistant  Secretary  of  Defense  (Logistics  &  Materiel  Readiness) 

ATAC 

Advanced  Traceability  and  Control 

BCA 

Business  Case  Analysis 

ERP 

Enterprise  Resource  Planning 

IPT 

Integrated  Product  Team 

NAVSUP 

Naval  Supply  Systems  Command 

PBL 

Performance-Based  Logistics 

PSI 

Product  Support  Integrator 

QMD 

Quantitative  Methods  Division 

USD(AT&L) 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 

WSS 

Weapon  System  Support 
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Whistleblower  Protection 

U.S.  Department  of  Defense 

The  Whistleblower  Protection  Ombudsman's  role  is  to 
educate  agency  employees  about  prohibitions  on  retaliation 
and  employees’  rights  and  remedies  available  for  reprisal. 
The  DoD  Hotline  Director  is  the  designated  ombudsman. 
For  more  information,  please  visit  the  Whistleblower 
webpage  at  www.dodig.mil/programs/whistleblower. 


For  more  information  about  DoD  IG 
reports  or  activities,  please  contact  us: 

Congressional  Liaison 

congressional@dodig.mil;  703.604.8324 

Media  Contact 

public.affairs@dodig.mil;  703.604.8324 

For  Report  Notifications 

http://www.dodig.mil/pubs/email_update.cfm 

Twitter 

twitter.com/DoD_IG 

DoD  Hotline 

dodig.mil/hotline 
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